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SUMMARY 



During the past year, our research efforts have concentrated on three areas: establishing a WAN ATM Token 
ring for tele-imaging, solving a missing link in PACS, and using PACS as a research tool. A WAN ATM 
OC-12 token ring has been installed at UCSF connecting six campuses in the San Francisco Bay area. In 
this ring, the daily clinical PACS and the research imaging network each has its own independent path to 
avoid interference. A missing link in PACS is digital mammography. Great emphasis in our research has 
been on the image acquisition, archive, display, and transmission of large digital mammogram files. PACS 
has matured as a clinical tool, but its applications in basic and clinical research has just begun. Using it as a 
tool, we have started research projects on image informatics, a digital atlas of hand images, teleconsultation, 
and tele-microscopic imaging. 

This report summarizes the current status of the development of high speed WAN, using PACS as a 
research tool, and the image informatics research in our laboratory during the past year. It contains 
preprints of papers to appear in the Proceedings of the International Society for Optical Engineering, SPIE 
Medical Imaging 1997, San Diego, California, February 21-27, 1998. This report also contains selected 
reprints and preprints from last year; abstracts and presentations from the 83rd RSNA, November - 
December, 1997. 
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ABSTRACT 

Picture Archiving and Communication Systems (PACS) will evolve into more network- 
centric, distributed systems in the near future. The reasons for this are better manageability, 
higher scalability and reduced cost of ownership. Medical image display software should be pre- 
pared for this development by making optimized use of networked resources, for instant large 
medical image files. Most software products currently used, however, is not designed for 
networked operation and imposes architectural limitations on the possible performance. We will 
propose an alternative software architecture for medical image display systems that can overcome 
this limitations without affecting the performance of conventional disk based I/O. 

Keywords: image display, server-centric computing, PACS architecture, network computer 



1. INTRODUCTION 

We currently observe a general trend towards server-centric computing in information tech- 
nology. The tremendous growth and success of the Internet, the global network connecting 
millions of computers, have triggered this evolution. Applications such as World Wide Web 
(WWW) browsers gather information that is scattered over various servers and present them in an 
integrated, consistent manner independent of the users physical location. One aspect of this tech- 
nology is that most of the application logic is moved from the client workstation to the network 
server. This has several benefits: The client workstation is only responsible for the presentation of 
the information and can therefore be implemented as a “thin client” with simpler technology and 
reduced maintenance costs. Implementing the application logic on the server can lead to better 
scalability and manageability 1 . 

This development will certainly have an impact on medical information technology (IT) 
systems and Picture Archiving and Communications Systems (PACS). Currently, administering a 
large-scale PACS is a challenging task, which becomes increasingly difficult as the number of 
client workstations grows due to the attempt to implement the filmless hospital 2 . One of the most 
complex problems is caused by a low-bandwidth network infrastructure, which forces a PACS to 
preload all images to the workstations where they will probably be needed 3 . With the availability 
of high-bandwidth networks such as ATM it becomes conceivable to implement image transfer 
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“on demand” where images are transferred to the workstation at precisely the moment when they 
should be displayed. While this approach would improve the manageability of the system, it 
imposes new challenges to the display software on the client workstation. 

Many medical image display applications operate only with images stored on the local hard 
disk. When images are located on a network server, they will first make a copy of the images to 
the local storage, and then display them. While this mode of operation will certainly work under 
most circumstances, it is not optimized for networked operation regarding speed and overall 
efficiency. 

Another aspect is that the increasing number of clients will cause large-scale PACS to evolve 
into distributed systems with one ore more archive servers and several departmental application 
servers. It likely that these systems will maintain connections to distant sites on a wide-area or 
even global scale. (Figure 1) 




Figure 1: Distributed PACS with three Application Servers (AS) and multiple Workstations (WS). 

Connections exist to Hospital Information System (HIS), Radiological Information Sys- 
tem (RIS), PACS Archive and several remote servers via Wide-Area Network (WAN) 

We should therefore expect that images might be located on different network servers spread 
throughout a local area or wide area network. Each of these servers might use another network 
protocol to access the images. They may use various image formats to encode the images. A 
medical imaging application should handle such a situation in an efficient, transparent and con- 
sistent manner. We will outline the architectural changes necessary to accomplish this goal. 



2. OVERVIEW 

At UCSF Laboratory for Radiological Informatics, we have developed a medical image display 
application optimized for networked operation. According to the described usage scenario, we 
defined a list of requirements for this application. The software should 

• display images transmitted from a network immediately, without storing them to a local 
disc first 
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• handle multiple network connections to different servers concurrently 

• use a variety of network protocols, including Hypertext Transfer Protocol (HTTP), File 
Transfer Protocol (FTP) and Digital Imaging and Communications in Medicine (DICOM) 
Protocol 

• decode mages in several image formats, including JPEG, TIFF and DICOM images 

• perform the network communications asynchronously, thus not blocking the user interface 
while receiving images 

• display images progressively when detecting a slow network connection. The user should 
be able to manipulate the part of the image that is already visible. 

• maximize throughput when working with a high-speed network 

• be independent of a specific image processing library 

• be reusable in other display applications 

To implement this functionality and to perform the necessary optimizations we developed a 

new architecture for medical image display applications that differs from conventional soft- 
ware in several ways: 

1. To support multiple network protocols, image formats and image processing libraries, we 
used a layered architecture that hides protocol and image format specific details and en- 
capsulates third-party libraries. 

2. To support progressive image display, we separated the user interface code and the net- 
work communication code into different threads. A thread-synchronization protocol 
ensures data integrity. 

3. To maximize network throughput and overall efficiency, we used the asynchronous 
Input/Output (I/O) model for the network communication 

. We will discuss these items in more detail in the next section. 



3. ARCHITECTURE DETAILS 



1. Layered Architecture 

For an application that communicates with multiple servers in a distributed system, it is 
essential to be open to all network protocols and file formats. We therefore used a layered 
architecture to implement the network transport for the display component. This scheme consists 
of a Protocol Layer, an Image Format Layer and an Image Processing Layer. (Figure 2) The 
Layers are isolated from each other and communicate through an internal, generic protocol. This 
architecture allows any combination of network protocol, image format and image processing 
library to work together. This design also makes it easy to extend the software with additional 
protocols, image formats or image processing libraries. 
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Figure 2: Layered Architecture of the network communication software 

We included support for multiple image processing libraries for two reasons: first, each of the 
libraries used has distinctive advantages and disadvantages. For instance, the DIMPL library is 
highly optimized to process grayscale radiological images, but lacks support for color images, 
which can be conveniently processed by the Intel Image Processing Library. Second, it can be 
useful to replace one image processing library by another for reasons of cross-platform portabil- 
ity. 

2. Multithreaded Design 

Displaying an image on a computer monitor involves several steps: (1) loading the image data 
into Random Access Memory (RAM); (2) performing image processing such as Lookup Table 
(LUT) transformation; and (3) transfer the resulting image into the computers video memory. 
Step (2) and (3) can then be repeated as the user modifies the display parameters. Conventional 
image display software performs these steps sequentially, one after the other 4 . During this 
procedure, no user input is accepted. When images are loaded from a local hard disk and the 
whole procedure takes only a very short time, this may be acceptable. However, the time required 
to receive an image from a distant network server can be unpredictable. If user input is ignored 
during that period, it gives the user the impression that the application has crashed. 

One way to avoid this problem is using a multithreaded design. Threads are subroutines within 
an application process that run independently and quasi-simultaneously. Our software uses two 
threads: one user interface thread that receives keyboard and mouse input from the user, and a 
network communication thread. Since both threads mn independent from each other, the user can 
continue to work while an image is downloaded from the network. 

Many advanced imaging applications use multiple threads to support image downloading in 
the background. Enabling progressive image display during the download, however, required us 
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to develop a thread-safe communications protocol between the network communication (NC) 
thread and the user interface (UI) thread (Figure 3). 

1. When an image should be displayed, the UI thread creates a new image object, which 
contains no information besides the data source locator that describes where to retrieve the 
image 

2. The UI thread sends a message to the NC thread indicating that an image should be 
downloaded. The UI thread draws the image to the screen, which is a null operation at this 
time. 

3. The NC thread locates the network server and begins downloading the image. Each chunk 
of data will be decoded when it arrives 

4. When the NC thread has decoded the header information, it sends a notification to the UI 
thread indicating the header is present. At this point, the UI thread adjusts the display 
parameters such as scale, brightness and contrast. If the header contains patient and study 
information, the UI thread can display the text overlay 

5. Whenever a portion of the image becomes available, the NC thread notifies the UI thread 
that new image data has arrived. Depending on the speed of the connection, the UI thread 
may choose to display that portion of the image immediately or to wait unit more data is 
ready. Only the newly arrived part of the image will be drawn to the screen. 

6. After completion of the transfer, the NC thread notifies the UI thread that the image is now 
ready. The UI thread draws any remaining part of the image to the screen. 




Figure 3: Illustration of the communications protocol between the User Interface (UI) thread, the 
Network Communications (NC) thread and the image object. 
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To avoid unnecessary screen redraws when transmitting an image over a high-speed network, 
we built a mechanism into the NC thread that detects the transfer speed and adjusts the rate of 
notifications sent to the UI thread automatically. We verified that this mechanism works over a 
high range of network speeds from modem connections to ATM. 



3. Asynchronous I/O 

Another important requirement for our software was that it should be capable of downloading 
images from multiple servers at the same time. The easiest way to implement this in the multi- 
threaded design would be to create a new thread for each server. The drawback of this solution is 
that it creates a race condition between these threads. A single-processor computer can only exe- 
cute a single thread at a time and simulates true multithreading by switching very fast between the 
threads. Since usually most threads are waiting for a certain input event, this is usually not very 
serious. However, when several threads are doing processor-intensive tasks such as image proc- 
essing, switching between threads can become very expensive and a waste of resources. 

In our case, this leads to the following effect: When a slow network connection is used, the 
time used to process the images is small compared to the time each thread waits for the data from 
the network. When the network speed is high, data is delivered faster than it can be processed, 
and the threads enter a race condition. This means that using one thread for each server causes a 
performance degradation that becomes worse when the network speed increases. Such a behavior 
is of course inefficient and limits the overall throughput of the application. 

We used the asynchronous (nonblocking) I/O model to overcome this problem. With asyn- 
chronous I/O, a single thread can maintain multiple network connections concurrently. To do so, 
however, we had to revert the flow of control and data within the application. Instead of 
“pulling” data out of the network and passing this request down the different software layers 
(Figure 4), data is “pushed” from the network and passed upwards through the layers (Figure 5). 




Figure 4: Data and control flow in the conventional, synchronous model using “pulling”. Note that 
this model can handle only a single image/connection per thread. 
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Figure 5: Data and control flow in the asynchronous model as used in the software component de- 
scribed in this paper. Data is “pushed” from the network to the higher levels. This figure 
shows also how a single thread can effectively handle multiple active server connections 
simultaneously with a simple dispatch mechanism. 

Unfortunately, not all of the software libraries used to implement our software support this 
mode of operation. For instance, the JPEG and TIFF image format libraries cannot use asynchro- 
nous I/O and thus require an additional thread. We are currently evaluating whether the possible 
performance gains justify the effort to rewrite these libraries. 

4. DISCUSSION 

The presented image display software implements several new concepts which can enhance 
the performance and efficiency when images are transferred from a network server and displayed 
immediately. We believe that high-bandwidth networks such as ATM and gigabit ethemet will 
enable future PACS to use this type of instant network-based display methods instead of the 
complicated prefetching methods commonly used today. Since our proposed display software 
architecture imposes no performance penalty when loading files from a disk, it could be 
incorporated into any medical image display software without drawbacks, besides the slightly 
higher complexity of the internal control flow. 

We implemented this software architecture first in an advanced radiological teleconsultation 
system 5 , which we currently test under clinical conditions. With the experiences gained from the 
clinical testing of the software, we will be able to refine the architecture. We are also performing 
extensive tests involving network connections with various speeds, different server operating 
systems and image sizes to determine the performance advantages of our software architecture 
under realistic conditions. Theses measurements will not only enable us to further improve the 
display architecture details, but will give us valuable hints for optimizing the server software 
architecture. 



5. MATERIALS AND METHODS 

We used the following tools and libraries for the described work: Microsoft Visual C++® 5.0, 
DICOM library of the University of California Davis Medical Center (UCDMC), JPEG library of 
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the Independent JPEG Group (DG), the TIFF library of Sam Leffler (Silicon Graphics Inc.), and 
the DOME Image Processing Library (DIMPL) of DOME, Inc. 
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Diagnostic Workstation for Digital Hand Atlas in Bone Age Assessment 

Fei Cao, H.K. Huang, E. Pietka, Vicente Gilsanz, Steven Ominsky 
Department of Radiology, University of California, School of Medicine, San Francisco, CA 94143-0628 



ABSTRACT 

Bone age assessment by a radiological examination of a hand and wrist image is a procedure frequently 
performed in pediatric patients to evaluate growth disorders, determine growth potential in children and 
monitor therapy effects. The assessment method currently used in radiological diagnosis is based on atlas 
matching of the diagnosed hand image with the reference set of atlas patterns, which was developed in 
1930s and is not fully applicable for children of today. We intend to implement a diagnostic workstation 
for creating a new reference set of clinically normal images which will serve as a digital atlas and can be 
used for a computer-assisted bone age assessment. The diagnostic workstation consists of a 200Mhz Sun 
Ultra 2 with 14 gigabyte fast-wide SCSI disks, one creator display board and one IK display monitor. 
The reference radiographs from CR system and film scanner are stored in the local disks of the diagnostic 
workstation for rapid image and data retrieval. User interface, image display and feature extraction 
software have been developed on this diagnostic workstation for creating the digital atlas. We’ll then 
develop a software package for the computer-assisted bone age assessment which will extract radiological 
findings from a patient image and allow for their comparison with corresponding values extracted from 
this digital atlas. By the end of this study there will be a medically accepted standard digital atlas. It will 
remove the disadvantages of the currently out-of-date one and allow the bone age assessment to be 
computerized and standardized. 

In this paper, we describe the initial data-collection and system setup phase of this five-year research 
program. We presents the system design, user interface and software tools for collection, visualization and 
management of clinical hand and wrist images for the reference atlas and concurrent comparison study. 

Keywords: digital atlas, pediatric radiology, hand image, skeletal age assessment, computers 



ACKNOWLEDGMENT 

This research is partially supported by a NIH Grant No. R01-LM06270. 



12 




Migration of the Digital Interactive Breast Imaging Teaching File 

Fei Cao 1 , Edward A. Sickles 2 , H.K. Huang 3 , Xiaoqiang Zhou 4 
Department of Radiology, University of California, School of Medicine, San Francisco, CA 94143-0628 



ABSTRACT 

The digital breast imaging teaching file developed during the last two years in our laboratory has been used successfully at 
UCSF (University of California, San Francisco) as a routine teaching tool for training radiology residents and fellows in 
mammography. Building on this success, we have ported the teaching file from an old Pixar imaging/Sun SPARC 470 
display system to our newly designed telemammography display workstation (Ultra SPARC 2 platform with two DOME 
Md5/SBX display boards). The old Pixar/Sun 470 system, although adequate for fast and high-resolution image display, is 4- 
year-old technology, expensive to maintain and difficult to upgrade. The new display workstation is more cost-effective and is 
also compatible with the digital image format from a full-field direct digital mammography system. The digital teaching file 
is built on a sophisticated computer-aided instruction (CAI) model, which simulates the management sequences used in 
imaging interpretation and work-up. Each user can be prompted to respond by making his/her own observations, assessments, 
and work-up decisions as well as the marking of image abnormalities. This effectively replaces the traditional “show-and-tell” 
teaching file experience with an interactive, response-driven type of instruction. 

Keywords: digital imaging; display; breast imaging; computer-aided instruction; education aid 



1. INTRODUCTION 

Routine screening with periodic mammography has been proved to be effective for early detection of breast cancer and in 
lowering its death rate[l-2]. Because of limited resources and facilities to provide hands-on instruction in breast imaging 
interpretation, there is a need to develop an alternate method to supplement or replace the current film-based teaching file 
method of training. A digital-based system can alleviate this shortage since a digital teaching file can be duplicated easily and 
disseminated widely, to be used simultaneously by many individuals at different sites for continuing medical education. This 
promises to improve the quality of breast imaging interpretation, and thereby facilitate mass screening with mammography 
and the problem-solving imaging evaluation of screening-detected abnormalities. 

During the last two years, we have developed a digital breast imaging teaching file on a 2K high-resolution display 
workstation which is built on the Pixar imaging/Sun SPARC 470 platform [3]. This digital teaching file has been used 
successfully at UCSF (University of California, San Francisco) as a routine teaching tool. It is available to all radiology 
residents, fellows, staff radiologists, and visiting radiologists for continuing medical education at UCSF. All radiology 
residents are now required to complete a set of 25 cases in the digital teaching file during each 4-week breast imaging rotation 
at UCSF. We also have incorporated use of the digital teaching file into several of the San Francisco-based postgraduate breast 
imaging courses given at UCSF. The digital teaching file currently contains 50 active cases, with approximately 500 digital 
mammograms and approximately 800 questions. It takes about 10 hours for an average user to complete a 50 case set. Dr. 
Sickles and his associates are constantly adding new cases, thereby replenishing the teaching file with new material. 
Mammograms of 200 carefully selected cases have been scanned in and maintained in digital format, from which active case 
sets will be updated periodically. 

Building on this success, we have ported the teaching file from an old Pixar imaging/Sun SPARC 470 display system to our 
newly designed telemammography display workstation [4] (Ultra SPARC 2 platform [Mountain View, CA] with two DOME 
[Waltham, Mass] Md5/SBX display boards). The old Pixar/Sun 470 system, although adequate for fast and high-resolution 
image display, is 4-year-old technology, expensive to maintain and difficult to upgrade. The new display workstation is more 
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cost-effective and is also compatible with the digital image format from a full-field direct digital mammography system. This 
system provides mammograms already in digital format, which can be selected and incorporated easily into the digital teaching 
file. 



2. THE PROCESS OF MIGRATION 

Migrating of a large clinic application, such as the digital teaching file, to a new display system is not a trivial task. A 
portability issue arises regarding different hardware, operating system, image format and display software. 

old 2K display station 

The old 2K mammography display station is sketched in Figure 1. The old 2K station is based on a Sun Sparc 470 
workstation, two 21" diagonal 2K MegaScan monitors, Pixar image processing hardware and a 7 GByte fast storage 
disk, packaged by Vicom Inc. (Fremont, CA) . 




Figure 1 . Configuration of old 2K display station. 

The Sun 470 workstation (64MB system memory) serves as a platform for the display station. Each of two gray 
scale monitors has 10 bit resolution and is able to display 2560 scan lines, each with 2048 pixels. The Pixar image 
processing system can directly access the digital image stored in the fast-disk. A 10MB digital mammogram takes 
less than 2 seconds to retrieve from the fast-disk and display on the 2K monitors. The Pixar hardware also comes 
with two 32MB frame buffers and optimized image processing routines. This makes it possible for four 
mammograms (<10MB each), in a typical display mode, to be easily manipulated on screen in nearly real time. The 
7 GB fast-disk is capable of storing 1,000 digital mammograms for fast retrieval by the Pixar system. 

new 2k display system 
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Figure 2. Configuration of new 2K display station. 

The new 2K telemammography display station is sketched in Figure 2. This display system was designed in our lab 
for viewing the 4K x 5.6K breast images from the full-field direct digital mammography system developed by 
Fischer Imaging Company(Denver, CO) [4], The display workstation uses the Sun Sparc Ultra-2 platform with one 
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200 MHz CPU and 256 MByte of memory, two DOME MD5-SBX boards with 8 MByte frame buffer, two 2K x 
2.5K Data Ray monitors (Westminster, CO), and Sun Enterprice3000 disk array (Mountain View, CA); 



The success of our digital teaching file is due to three factors: (1) high-resolution display which is able to show an entire 
breast with such fine detail that tiny structures are readily visible; (2) sophisticated computer-aided instruction (CAI) model in 
simulating an interactive teaching session, especially allowing the user to identify abnormalities by marking displayed 
images, which is one of key elements in breast imaging interpretation. (3) large set of cases, selected and prepared by expert 
radiologists, and easy to maintain and replenish. All of those involve with the image display and processing, which is quite 
different for the old and new display systems. 

Following is the portability table summarizing the process of migration we have completed from the old Sun470/Pixar 
imaging system to the new Ultra 2/DOME MD5-SBX display system. 





Old System 

Sparc470/Pixar/FD 


New System 

SparcUltra2/Dome board 


Compatible 


Operating System 


SunOS4.1.3 


Solaris 2.5.1 


no 


User Interface 


Xview3.0 


Xview3.2 


yes 


Coding 


C 


C 


yes 


Image Display & 
Processing 


Pixar Chap Lib 


XIL1.3 

(Sun X Image Lib) 


None 


Digital Image Format 


Pixar picio tile 
ACE-NEMA2.0 


Raw & 
DICOM 


no 


Teach File Text 


plain text 


plain text 


yes 


ROI Data 


binary 


binary 


Conversion 



Table 1. Porting guide 

Much of our porting efforts are in converting the image display and processing software from the special Pixar image library 
to XIL (X Image Library). The XIL library is an imaging API (Application Program Interface) that provides a broad set of 
functions for imaging and video applications. The XIL library is part of a strategy for low-level software interfaces 
(foundation libraries) and enables imaging library and imaging application developers to port their code to these foundation 
libraries. The XIL library is hardware-independent image processing library which is MT-hot and fully supported by Sun and 
is also compatible with DOME MD5-SBX board. The image formats are also converted from the Pixar-specific picio tile and 
old ACE-NEMA2.0 standard ones to Raw and current DICOM standard formats. The porting from the old to the new display 
system has made our digital teach file up to date with current standards and technologies, which makes it possible to maintain 
and future upgrade the digital teaching file. 

comparison of image display quality and speed 

A mammography display has to be able to portray the entire breast with such fine detail that tiny structures are readily 
visible. This must be done at near real-time speed. The quality of digital mammograms is highly dependent on the acquired 
image resolution and display system hardware, while the speed of image display and on-screen image manipulation is 
determined by system hardware architecture and image processing software. 

It is noted that the image display and manipulation on the old Pixar system is faster, and the image quality is also better than 
the DOME display boards. The faster imaging speed on the old system is due to its special hardware design. Pixar image 
hardware with two large frame buffers (2 x 32 MByte) and optimized processing software makes the fast on-screen image 
manipulation possible. The transfer of image data between disk storage and the image computer accounts for a substantial 
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percentage of the total time consumed during an imaging job. The principal difference between the Fast-Disk subsystem used 
in the old display workstation and a conventional hard disk is the high data throughput rate. The Fast-Disk subsystem 
achieves its speed by distributing each input/output operation over the four drives within the system. In addition to the high 
data transfer rate, the Fast-Disk subsystem also operates in conjunction with the Pixar imaging system to deliver data directly 
to the frame buffers without a layover in the memory of the host . In comparison, the two DOME MD5-SBX boards come 
only with two 8 MByte frame buffers and use the host computer’s memory for image transfer and manipulation. Preliminary 
results indicate that a 10-MByte digital mammogram takes less than 2 seconds to retrieve from the Fast-Disk and display on 
the old Pixar system while it takes about 4 seconds for the new Ultra2/DOME system. The lObit display resolution on the 
Pixar system is also superior to the Dome’s 8bit resolution display. 

However, the display speed and the image quality from the new Dome MD5-SBX boards are found to be adequate and 
acceptable for a clinical application such as our digital teaching file. Furthermore, the new display system costs much less, 
and is much easier to maintain and upgrade than the old Pixar system. 



INTERACTIVE DIGITAL BREAST IMAGING TEACHING FILE: AN EXAMPLE 

The implementation method and the CAI model of the digital teaching file have been fully described in our online paper[3]. 
The CAI model simulates clinically relevant work-up sequences and provides an interactive learning environment. 
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Figure 3. A general algorithm for the CAI model and basic work-up sequences 



Figure 3 shows the general control flow of this model and imaging work-up sequences. At the start of each case, the system 
first gives a brief description of the case history and then presents to the user a set of initial screening mammograms. After 
examination of those images, the user is asked to use a mouse pointer, identifying on the 2K monitors any mammographic 
abnormalities. The user’s selection is compared with pre-recorded region of interest (ROI) data. If the user succeeds in 
identifying the ROI, he/she will proceed to a multiple choice or true/false question that is relevant to his/her findings, or the 
user must try again. If a user has not marked the abnormality after three tries, the system will display the ROI on the 2K 
monitor, accompanied by an explanation; the user then proceeds to the next question. Based on user response to a question, 
the computer system presents different work-up sequences, defined in the CAI model as illustrated in Figure 3. The system 
provides instructions, explanations, and answers to the question, guiding the user toward the next question. During this 
process, additional images are displayed whenever appropriate. This control flow serves, in a structural way, to aid the user in 
navigating through the medical knowledge embedded in the teaching file cases. 



an example 

Following is an example of Case 1 in the digital teaching file. The breast images are shown on the two 2K monitors, while 
the questions and answers are displayed on the Console monitor 

CASE 1. 46 year old asymptomatic woman undergoing baseline screening mammography. 

The next set of questions will ask you whether there is an important mammographic finding on each of the 4 
displayed images, and if so, to indicate the location of that finding. Press the Question button once you have fully 
evaluated the mammographic images. 



16 








17 







For an example, select the mammographic findings on CC and MLO views of the right breast. 

Press the ROI-Select button and then use the mouse to point at and select the one mammographic finding that is 
MOST important on this CC view of the right breast. If there is no important finding, press the middle mouse 
button. 

The system will display one of the following messages on screen, corresponding to a user’s selection 

INCORRECT. You have not selected the most important mammographic finding on this view, or there is no 
important finding. CHOOSE AGAIN. Press the question button to repeat this question. 

CORRECT. There is a noncalcified density in the medial aspect of the right breast on this CC view. Press the 
Question button to continue to the MLO view of the left breast. 

You have still not selected the most important mammographic finding. For your assistance, the finding is now 
highlighted on the monitor. Press the Question button to continue to the next question. 

Then continue to the MLO view of the right breast as shown in the next figure with the finding highlighted on the left 
monitor and a zoom-in box on the right monitor. After the user has identified the abnormalities on each of the four images, 
the CC and MLO views of the right breast are shown on the two 2K monitors and the user is then asked to answer the 
following multiple choice question regarding the mammographic abnormality 

Which one of the following choices BEST describes the mammographic abnormality in this case? 

A. Focal asymmetric density 

B. Circumscribed carcinoma 

C. Architectural distortion 

D. Noncalcified mass 

E. Fibroadenoma 

For the user’s choice, such as (A) or (D), the system will response with 

A. A focal asymmetric density is defined as an asymmetry of fibroglandular-tissue density with similar shape on 
two views, but lacking the convex-outward borders and conspicuity of a true mass. While such a finding may 
represent an island of normal fib ro glandular tissue, its lack of specific benign characteristics may warrant further 
evaluation. However, the images provided do not portray a lesion that fits the criteria that define a focal asymmetric 
density. CHOOSE AGAIN. Press the Question button to repeat this question. 

D. CORRECT. A mass is a space occupying lesion seen on two projections, with convex-outward borders, usually 
denser in the center than at the periphery. Press the Question button to continue to the next question. 

Two more follow-up questions 

Which one of the following choices BEST describes the location of the mammographic abnormality in this case? 

A. Upper inner quadrant 

B. Upper outer quadrant 

C. Lower outer quadrant 

D. Lower inner quadrant 

Which one of the following choices indicates the MOST appropriate next step in the imaging work-up? 

A. Surgical biopsy 

B. Follow-up mammography in 6 months 

C. No additional studies are necessary (likely diagnosis, intramammarylymph node). 

D. Ultrasonography 

E. Correlation with physical examination 
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Then standard TRUE/FALSE question 

The following four questions are TRUE/FALSE questions concerning the use of breast ultrasonography. 

Ultrasonography is important in the work-up of lesions seen on mammography because it reliably distinguishes 
between cystic and solid masses. (TRUE/FALSE) 

If the user’s answer is TRUE 

You answered TRUE. This is CORRECT. The diagnostic accuracy of ultrasonography in distinguishing simple 
cysts from solid and complex cystic lesions approaches 100%, if rigid interpretive criteria are utilized. For reliable 
characterization of a mass as a simple cyst, it should show smooth and well-defined margins, increased through 
transmission of sound, and be completely anechoic. Press the Question button to continue to the next question. 

Next three TRUE/FALSE questions and then followed by 

Based on the displayed sonographic images, which one of the following choices indicates the MOST appropriate next 
step in the work up? 

A. Aspiration under sonographic guidance, followed by pneumocystography 

B. Surgical biopsy 

C. Doppler ultrasound examination 

D. Spot-compression magnification mammography 
If the user makes a right choice (D), the system will response with 

D. CORRECT. The mass is presumed to be solid since it does not display the full complement of sonographic 
features characteristic of a simple cyst. Spot-compression magnification mammography will provide improved 
definition of the margins of the mass, and thus may guide management recommendations. Press the Question 
button to continue to the next question. 

Question 9. Based on the displayed additional spot-compression magnification view, which one of the following 
choices indicates the MOST appropriate management recommendation for this case? 

A. Needle localization prior to open surgical biopsy 

B. Percutaneous image-guided core biopsy 

C. 3 month follow-up mammography 

D. 6 month follow-up mammography 

E. Routine mammographic screening in one year 

For the correct answer, the system will response with 

D. CORRECT. The probability of carcinoma for a noncalcified, circumscribed, solid mass is only 1% to 2%. This 
would result in the need to excise or obtain tissue samples from 50 to 100 lesions in order to identify only one 
cancer. These procedures are more invasive and more expensive than periodic mammographic surveillance. 
Furthermore, two independent large-scale prospective studies of long-term mammographic follow-up of such 
"probably benign" masses have shown that those few lesions which actually are malignant indeed are correctly 
diagnosed by interval mammographic change while they still remain curable and amenable to treatment with breast 
conservation surgery. Indeed, the major prognostic factors (tumor size, axillary lymph node status, and tumor stage) 
of such cancers, although excised months to years after they first were detected, are just as favorable as those for 
screening-detected cancers. 

*** EPILOG *** -> The patient returned in 6 months and the finding was stable in appearance (films not shown). 
The finding is therefore even more probably benign than had been thought before. Continued mammographic 
surveillance is needed, at lengthening intervals. Most radiologists would stop requesting mammographic 
surveillance after two or three years of stability. Following such a protocolreduces both morbidity and cost by 
substantially decreasing the number of open surgical and percutaneous biopsies performed for benign masses. It does 
not compromise the identification of early-stage, favorable-prognosis cancers. 

END of Case 1. Press the Question button to begin the next case. 
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SUMMARY 



We have successfully ported the digital teaching file from the old one with the SunOS/Pixar imaging system to the new one 
with the Solari s/DOME board display system. The new display system is compatible with the current display technology, and 
also more affordable and easy to maintain. Especially, the large quantity of breast images have been converted to the current 
standard format, and the image display and processing software have been ported to the fundamental XIL library which is 
hardware-independent and available for free. This migration from the old system to the new one not only insures maintaining 
and replenishing of our current digital teaching file, but also makes it possible to distribute the UCSF digital teaching file to 
other institutions, in an affordable price, for training radiologists in breast imaging. 
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ABSTRACT 

Surgical treatment of patients suffering from complex partial seizures requires the localization of the epileptogenic 
zone for surgical resection. Currently, clinicians utilize electroencephalography (EEG), psychological tests, and various 
neuroimaging modalities together to determine the location of this zone. We investigate the use of positron emission 
tomography (PET), magnetic resonance imaging (MRI), and magnetic resonance spectroscopy (MRS) in the presurgical 
workup and analysis of patients with complex partial seizures. The results of imaging studies of 25 patients are compared for 
lateralization accuracy and relative concordance. 



1. INTRODUCTION 

Epilepsy refers to a tendency toward recurrent seizures, i.e., paroxysmal derangement of cerebral function due to uncontrolled, 
excessive discharges from an aggregate of neurons. As a population, patients with epilepsy are undereducated, underemployed, 
and have a life span approximately 10 years less than their peers 1 . The total cost of epilepsy taking into account 
unemployment, underemployment, excess mortality, long-term treatment, residential care including institutionalization, costs 
of drugs, vocational rehabilitation, and special education at the time of the epilepsy commission, is estimated at between 3 
and 4 billion dollars per year' 1 . 

Despite the advances made in anti-epileptic drugs, about 25% of epilepsy patients have seizures uncontrolled by 
medical treatment, i.e., medically refractory. Approximately half of these patients are candidates for surgical treatment. 
Surgical planning of epilepsy requires the accurate identification of the area of the brain responsible for seizure onset, known 
as the epileptogenic zone. It has been estimated that approximately 125,000 patients of medically refractory epilepsy in the 
United States may be suitable candidates for surgical treatment^, and an estimated 5,000 per year are added to this patient 
population. 

The importance of surgical treatment was recently confirmed by an NIH (National Institutes of Health) consensus 
conference^. The best surgical outcomes are obtained when the various preoperative localizing examinations, e.g., video 
electroencephalography telemetry (VET) and magnetic resonance imaging (MRI), are concordant, implicating the same brain 
region as the epileptogenic zone. Unfortunately, a number of patients have either discordant or non-conclusive imaging data. 

Currently, intracranial electroencephalography (EEG) recording is the gold standard for evaluating surgical candidates 
of medically refractory epilepsy. An intracranial EEG study is an invasive surgical procedure which requires placing electrodes 
directly on the surface of the brain or within the brain itself for recording brain electrical activity during spontaneous seizures 
to determine where they originate. This array of electrodes remains in the patient from a week to a few weeks for seizure 
onset monitoring. There are definite risks (6% to 8% morbidity rate) and considerable costs associated with such an invasive 
procedure ($40,000 to $80,000 per study which includes intracranial EEG monitoring in the hospital). 

These invasive and expensive procedures are not performed unless the results of noninvasive studies permit 
formulation of a reasonable guess about the possible lateralization or location of a resectable epileptogenic region. Usually, 
the noninvasive presurgical evaluation must identify lateralization choices before invasive testing can be planned. 



For further author information - 
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Technological advances in medical imaging bring forth many new noninvasive techniques that not only help to 
lateralize the epileptogenic region with reasonable certainty before invasive EEG recording is performed, but can also 
eliminate or reduce the frequency of such invasive testing. The technological revolution began with positron emission 
tomography (PET) scanning and the ability to display regional metabolic dysfunctions associated with epileptogenic zones. 
MRI followed, providing unsurpassed anatomic detail to detect structural abnormalities. Research studies utilizing PET and 
magnetic resonance spectroscopy (MRS) separately have demonstrated noticeable metabolic abnormalities in the epileptogenic 
zone. 



Multiple imaging methods are available to aid in lateralizing complex partial seizures. To reduce the need for 
intracranial electrodes and to localize the epileptogenic zone with a higher degree of confidence, 2-[18F]fluoro-2-deoxy-D- 
glucose (FDG) PET and MRI are used to provide confirmatory localizing information. Relative temporal lobe (TL) 
hypometabolism on FDG-PET and hippocampal atrophy or prolonged T2 on MRI reliably predict lateralization of the 
epileptogenic zone. 4 ’ 5 FDG-PET may demonstrate diminished glucose uptake associated with epileptogenic substrates even 
when the MRI is unremarkable. MRI-PET findings, however, may be non-localizing when structural and metabolic 
disturbances are widely distributed or equivocal, or when no abnormality is revealed. ^ H-MRS has been used to detect 
unilateral and bilateral metabolite abnormalities in patients with epilepsy. 6 Unlike EEG, which provides direct evidence of 
the origin of seizures, MRI, PET, and 1 H-MRS provide only indirect evidence of potential epileptogenic zones. 
Unfortunately, scalp-recorded EEG does not always yield clear localization of the epileptogenic zone either, and the 
relationship of EEG-detected abnormalities to structural lesions may be ambiguous. The significance of many structural and 
functional imaging abnormalities cannot be interpreted in isolation. 

This study compares FDG-PET, MRI, and 1 H-MRS in patients suffering from complex partial seizures. The high 
resolution FDG-PET is compared with quantitative MRI techniques and 1 H-MRS in the lateralization of the epileptogenic 
zone. The analysis of imaging data is carried out on a computer-assisted diagnostic workstation developed in our laboratory. 



2. DIAGNOSTIC WORKUP OF EPILEPSY 

Figure 1 outlines the general diagnostic workup for presurgical epilepsy diagnosis 7 . A patient who enters this workup is 
medically refractory as determined by patient history suggesting partial epilepsy and failure of drug treatment. EEG and MRI 
are routinely used to classify seizures and localize the epileptogenic zone. MRI provides visualization of brain anatomy with 
detail greatly suipassing that of any imaging modality, including x-ray computed tomography (CT). The spatial resolution 
provided by MRI has greatly improved the detection of most mass lesions or tumors. Detection of structural lesions in 
epilepsy makes surgical evaluation relatively straightforward. Many epilepsy cases, however, do not exhibit structural 
abnormalities in MR images. Further, paroxysmal abnormalities occur intermittently and may not be easily captured in EEG 
recordings. Prolonged inpatient recording with video EEG telemetry (VET) is necessary to capture spontaneous seizures. 
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Figure 1. Diagnostic workup of epilepsy surgical candidates. Neuroimaging modalities are being applied to noninvasive 
surgical planning of patients with complex partial seizures in hopes to minimize the need for invasive intracranial EEG tests; 
this contributes major savings in cost, reduced patient risks, and increased center capacity. 

Positive results from MRI and VET in localizing the epileptogenic zone typically indicates an outcome of becoming 
seizure free by surgery. Non-concordance between MRI and VET almost invariably requires invasive EEG recordings with 
either one or more of the following: subdural strips, arrays, and stereotactically inserted depth electrodes. For example, 
approximately 35% to 40% of medically refractory epilepsy patients in the Northern California Comprehensive Epilepsy 
Center operated on at the University of California at San Francisco (UCSF) require invasive intracranial EEG monitoring. 
The average cost of intracranial EEG is about $60,000 per evaluation and amounts to about $2 million to $3 million per year 
in the UCSF Epilepsy Center alone for performing this pre-operative procedure. 

The UCSF Epilepsy Center is currently performing additional imaging examinations, i.e., ^H-MRS, and PET, on 
selected epilepsy patients before coming to invasive evaluation. These additional imaging tests cost about $3,000 to $4,000 
per patient, a fraction of the cost (< 10%) of an intracranial EEG study. All neuroimaging studies of these patients are 
collected for computer analysis. The use of combined noninvasive tests to reduce the number and improve the accuracy of 
intracranial EEG recordings will help to reduce the cost of presurgical evaluation, minimize patient risk, and improve surgical 
outcome. In addition, a better understanding of decision thresholds from multimodal imaging analysis should improve the 
selection of candidates for surgery by increasing the identification of patients with previously unlocalized seizure origination. 

Nevertheless, an effective means to access, manipulate, and correlate large, heterogeneous multidimensional medical 
images was lacking. We thus developed a computer-aided diagnostic (CAD) workstation for epilepsy surgical planning to 
fulfill this need. This medical workstation aims to provide an environment for optimal interpretation of multiple non- 
invasive image modalities by combining functional information of PET and ^H-MRS with structural MRI anatomy, both 
qualitatively and quantitatively. Since there is currently little knowledge about the patient populations with positive and 
negative findings of various imaging and intracranial EEG studies, the data extracted will also be essential for outcome 
analyses to identify inefficient or duplicative evaluation procedure. 



3. INFRASTRUCTURE FOR MULTIMEDIA DATA MANAGEMENT 

Our computer-aided neurodiagnostic workstation system is built on top of the existing picture archiving and 
communication system (PACS) infrastructure at UCSF. PACSs have been the prevalent means for acquisition, storage, 
communication, and display of digital images related to radiological practice during the last decade®. A PACS integrates 
image acquisition devices, computers, communication networks, image display workstations, and database management 
systems. Its operations can be summarized as follows: (i) Digital images generated in an examination are captured by an 
acquisition computer and sent immediately to a central database for long term archive; (ii) these images then will be retrieved 
later from the central archive and transmitted to display at a remote medical display station automatically or upon the user's 
request; and (iii) diagnostic findings of viewing these images are appended to the images at the central archive. 

4. COMPUTER ASSISTED SURGERY PLANNING 

In Figure 2, we show the distributed system architecture of the CAD workstation together with the computer-aided 
workup steps that the workstation supports. The medical workstation provides a multimedia user interface to: (1) access 
medical data stored in the PACS file archive and the neuroimaging database and (2) analyze the retrieved data for noninvasive 
surgical planning of epilepsy®. For security reasons, all user queries from non-daily clinical operations pass through a 
multimedia data file server, which acts as a "firewall" against unauthorized access to the PACS central archive. All computers 
are SUN SPARC workstations running SUN OS, excepting the neuroimaging database server that resides in a 4-processor 
SUN SPARC 690 MP running Solaris Operating System. The multimedia user interface of the medical workstation is built 
on top of the Gain Momentum software package (Gain Technology Division, Sybase, Palo Alto, CA) and X window 
programming environment. Currently, not all the brain imaging modalities involved in epilepsy surgical planning have been 
archived into UCSF PACS. Nuclear medicine PET image datasets are manually archived into the neuroimaging database 
using the File Transfer Protocol (FTP). The neurodiagnostic workstation includes the routing table on which to retrieve these 
PET image files upon user query. 

Image registration and combination. Heterogeneous medical data retrieved from PACS or neuroimaging database are 
combined and rendered into 2D or 3D images. The visualization work is done in cooperation with the VIDA (volumetric 
image display and analysis) project at the University of Iowa Medical College. Before any data combination and analysis, 
however, medical image registration is often required to geometrically align two or more brain image volumes so that voxels 
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representing the same underlying anatomical structure may be superimposed or directly compared. A pair of image volumes 
might be from the same subject and the same modality (but taken at different positions or different scanning instruments), 
from the same subject but different modalities, or from the same modality but different subjects. 

Our registration strategy in this multimodal image fusion application is to have the high resolution, structural MRI 
dataset as the reference point for mapping across different imaging modalities. The MRI-MRS registration is straightforward: 
The MRS scans are performed immediately after the MRI examination of the patient using the same MR scanner (GE Signa 
or Siemens Vision) such that there is no change of the patient's head position. For the MRI-MRI, PET-PET, and MRI-PET 
image pairs, we implemented the technique of ratio image uniformity (RIU) described by Woods 10 . For the RIU technique, 
the translation and rotation parameters were determined such that they minimized the voxel-by-voxel variance of the 




"ratio" « w 

Figure 2. (a) Distributed system architecture of the computer-aided neurodiagnostic workstation, and (b) the computer-aided 
workup for epilepsy surgical planning that the medical workstation supports. 

image. The MRI volumes were stripped to remove the skull and scalp (the removal process is done semi-automatically) prior 
to calculating the ratio image for MRI-PET registration. RIU algorithms then iteratively minimize the ratio image variance 
using a modified Newton-Raphson method based on explicitly computed derivatives of the variance with respect to each of the 
registration parameters 10 . Recent study confirms that registration optimization based on measurement of the similarity of 
spatial distributions of voxel values is superior to techniques that do not use such information 1 

Extraction of structural and functional information. Once image pairs are registered into common space, their 
biomedical information can be combined, extracted, and quantitated to obtain important information about the location and 
nature of the epileptogenic zone. For example, the high resolution MR image provides a basis to anatomically map glucose 
metabolism. This allows the ability to identify metabolic rates in specific brain regions of interest. The neurodiagnostic 
workstation incorporates a program developed at Lawrence Berkeley Laboratory (Center of Functional Imaging, LBL) in 
conjunction with VIDA (VIDA Project, University of Iowa Medical College) to map regions of interest defined on the MRI 
to the PET sinograms for quantitation of glucose uptake. Figure 3. a illustrates such an example taken from another partial 
epilepsy case. This figure marked down the values obtained from structural/functional image analysis on volumes and 
18 FDG glucose uptake per unit volume of hippocampus and amygdala. Note that original MRI and PET images are shown 
in this figure. Such quantitative data, which are not obtained in conventional workups, can provide sensitive and objective 
localization information. 
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Figure 3. (a) Quantitation of hippocampus and amygdala MRI regions and the use of MRI-PET co-registration to obtain FDG 
concentration per unit volume in FDG-PET (2 dimensional original slices shown here), (b) MRI-MRS coregistration and 
spectral analysis. ifl-MRS spectra for the defined voxel region (white squares MR images) in the right temporal lobe as seen 
on the MRI. The peaks measured with MRS in this case represent choline (Cho), creatine (Cr), and N-acetyl-aspartate 



(NAA). 



Image query by contents. The clinician can save the analyzed results of the workup into the neuroimaging database for 
future reference. For example, structural and functional information extracted as shown in Figure 3. a and 3.b will be stored 
into the neuroimaging database to enable flexible user query of multimedia medical data. Figure 4 illustrates a graphical user 
interface for accessing information contained in the neuroimaging database. Query is by image content, i.e., MRI 
hippocampal volume or PET glucose concentration, or by patient attributes, i.e., age. In this example, three pairs of sliders 
that specify the range of the three aforementioned patient attributes. The image database returns with a list of patients 
satisfying the combined attribute ranges. The user can select any patient name to retrieve and display MRI and PET images 
(only two slices are shown here, and the PET image is registered to MRI image). The multimedia user interface is object- 
oriented and compositional, such that another interface format, tailored for different applications, can be rapidly implemented 



27 






i p ip 



F 36 8/1 S&4 32S21333_1964JJ8.1S SEIZURE BRAIN 58 47 

F -41 1 0/25/94 321 78fiG9_1984.10.25 SEIZURE BRAIN 50 47 

M 26 2/1495 3296644 J995XJ2.1 4 SECURE BRAIN 52 47 

(STp“ )8/3«S. j^^75JDCe7)8M^ S E EURE BRAI N~ js4~ ~J47~ 

M 45 12/2/94 0482672J994.12.02 SEEURE BRAIN 51 47 

F 24 5/23/95 0688107 J99SJQ523 SEEURE BRAIN 48 47 

M 21 11/20/94 129105t J994.il 20 SEEURE BRAIN 56 47 

M 45 2/6«5 32fi5T4BJH85BZJ0fi SEEURE BRAIN 56 4/ 



Summers. Nancy 
Pefie*Ariepe , 
Ganges; Michael 
jlloc. David 
iTolamdLVlJay 
Fenton,: Marina ; 
j Dodge. Jon 
\ Shepherd. OavM 






sj* use vicget: and type in itetds 



View MRl View PET 



Figure 4. Extracted image features can in turn be used for indexing the neuroimaging database, for example comparison and 
reference during surgical planning and diagnostic workup of a new case. 



using the same set of graphical icons and tools. In addition, as the image database grows, the analyzed data accumulated will 
be important in performing outcome analyses and in evaluating the cost-effectiveness of various imaging and surgical 
protocols. 



5. METHODS 



Surgical candidates suspected to have seizures arising from the temporal lobes were screened for inclusion in the prospective 
study. Informed consent was obtained from all subjects. Diagnosis of temporal lobe epilepsy was based on scalp-sphenoidal 
EEG/video telemetry. Of the 36 patients diagnosed to have epilepsy, 8 were excluded from the study due to foreign tissue 
lesions, including vascular malformations, glial tumors, hamartomas, and dysembryoplastic lesions. One patient was 
excluded because both J H-MRS and FDG-PET data sets were severely obscured by motion artifact. Two additional patients 
were excluded because they did not complete protocol examinations for either ^-MRS or hippocampal volumetry (HV). 

Ultimately, 25 patients (12 male, 13 female, age range 14-56, mean 38 years) were included in this study. Because 
T2 relaxometry did not begin at the start of this project, only 13 patients (6 male, 7 female, age range 14-56, mean 37 years) 
were studied with all four modalities. Twenty four patients have had surgery in which a tailored subpial temporal lobe 
resection was performed. All resected tissue demonstrated mild to moderate hippocampal gliosis, a feature of mesial temporal 
sclerosis, the most common pathology found in complex partial seizures. 

FDG-PET. PET scans were performed with a CTI/Siemens 961 HR EXACT scanner. Forty-seven slices were acquired 
simultaneously over a 15 cm field of view. Axial, coronal, and sagittal planes were reconstructed for visual interpretation. 

Diagnostic MRL All patients underwent diagnostic MRI studies independent of research MR and *H-MRS exams. 

UCSF MRI exams are performed at 1.5 T (Signa, General Electric) and the following sequences were acquired: sagittal T1 
weighted image (TR 600 ms/TE min) through the hippocampus and temporal lobes; axial T2 weighted (TR > 2000 ms, TE 
35/80 ms); 3-D SPGR gradient echo (TR 50 ms, TE min, flip angle 40 deg., slice thickness 1.5 mm); coronal T2* gradient 
echo (TR 500 ms, TE 15-34 ms, flip angle 20 deg.); and high resolution (512 x 512 matrix) coronal T2 fast spin echo (T2- 



28 





FSE) sequence. Interpretation of mesial temporal abnormalities was based on signal, morphological, and volume changes. 
Abnormalities of hippocampal internal structure were determined from the high-resolution T2-FSE sequence. 

Hippocampal volumetry (HV). All 25 patients had an MPRAGE sequence (TR/TE 10/4 ms, flip angle 15 deg, slice 
thickness 1.5 mm) acquired in the coronal plane orthogonal to the long axis of the hippocampus for HV in the same sitting 
as T2 and ^H-MRS exams. 



T2 relaxometry. Thirteen of the 25 patients had hippocampal T2 measured using a procedure similar to that reported by 
Jackson et al. and Grunewald et al. 12>13 A multi-spin echo sequence (16 echoes) with TR 2000 ms and TE 22.5, 45. ..360 
ms from an 8mm thick slice of the anterior hippocampus was performed. Two independent operators placed circular regions 
of interest (ROI) within the margins of the hippocampal body. The average of the two T2 relaxation times was calculated to 
obtain a final measurement for T2 of the corresponding hippocampus. 

1 H-MRS. Twenty four of the patients were studied with *H-MRS. These spectra were acquired on a 1.5 T Magnetom 
Vision"^ (Siemens, Erlangen, Germany) using a standard circularly polarized head coil according to the method previously 
reported by Ende et al. 14 Quantitation of the metabolites was based on use of the unsuppressed water signal obtained in a 
second MRS examination with otherwise identical measurement parameters. The concentration of NAA were used to 
determine metabolite abnormality and lateralization with *H-MRS. 

Image interpretation and analysis. Working on the CAD workstation, the PET images were interpreted for evidence 
of focal hypometabolism by two investigators. Five temporal lobe ROIs were selected - temporal pole, anterior medial, 
posterior medial, anterior lateral, and posterior lateral cortex - and scored for relative hypometabolism. the sum of the five 
regions was used to provide a qualitative measure of the degree of hypometabolism. An asymmetry index (Al) was used to 
measure the degree of asymmetry for HV, T2 mapping, ^H-MRS: 



! right + left 



( 1 ) 



6. RESULTS 

Twenty-three of 24 patients studies with PET had one or more regions with relative hypometabolism confined to a single TL. 
MRI revealed easily discernible relative hippocampal atrophy in fifteen patients. T2 hyperintensity was seen only with 
concurrent hippocampal atrophy. All cases of MRI identified hippocampal atrophy were concordant with ictal EEG. 
Seventeen of 25 patients were lateralized with HV. Nine of 13 patients were lateralized with T2 relaxometry. ^H-MRS 
lateralized 15 of 24 patients with once discordant lateralization in one patient. The results of each modality's ability to 
lateralize the epileptogenic zone are summarized in Table 1: 



modality 


lateralized 


n 


% 


Al 


PET 


23 


24 


96% 


NA 


MRI - hippocampal atrophy 


15 


25 


60% 


NA 


MRI - T2 increase 


9 


25 


36% 


NA 


HV 


17 


25 


68% 


>8% 


T2 relaxometry 


9 


13 


69% 


>4% 


*h-mrs 


15 


24 


63% 


> 12% 



Table 1. Lateralization of epileptogenic zone by neuroimaging modalities. 
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FDG-PET was the most sensitive imaging methodology for lateralizing complex partial seizures in the temporal lobe, 
lateralizing all but one patient with hippocampal atrophy and 75% of patients without atrophy. HV lateralized 68% of 
patients, and T2 relaxometry did not provide additional lateralizing information beyond that provided by HV. 

Comparisons of lateralization can only be done in those patients who completed all exams included in the evaluation. Table 
2 lists the lateralization from the 23 patients who completed FDG-PET, HV, and ^ H-MRS studies (from the total of 25 
patients, one did not have a PET, and in another the 1 H-MRS volume selection owas displaced above the plane of the 
hippocampi). No discordance was found with FDG-PET or HV. One patient was discordant with 1 H-MRS measures. 



modality 


concordant 


discordant 


non-lateralized 


FDG-PET asymm. score > 3 


20 


0 


3 


HV AI > 8% 


15 


0 


8 


1 H-MRS [NAA] AI> 12 


15 


1 


7 



Table 2. Comparison of lateralization (n=23) 



7. CONCLUSION 

The surgical planning of medically refractory epilepsy often requires invasive, intracranial EEG recordings to localize 
the epileptogenic zone for resection. There are, however, definite risks and considerable costs associated with such invasive 
procedures. Planning for surgery in epilepsy requires analyzing and combining structural and functional data from numerous 
sources to localize the seizure foci for resection. This reported work shows the results of studying the lateralizing accuracy of 
multiple neuroimaging modalities, e.g., MRI, PET, and 1 H-MRS, used in noninvasive planning of epilepsy surgery. Image 
analysis including ROI segmentation, ROI quantitation, and image registration was carried out on a CAD workstation. FDG- 
PET was the most sensitive imaging modality for lateralization in this study. FDG-PET was even able to lateralize the 
abnormality even in the absence of HV and other MRI evidence for hippocampal sclerosis, including T2 relaxometry , 
supporting the continued role for FDG-PET in the evaluation of patients with complex partial seizures in the temporal lobe. 

1 H-MRS was not as sensitive as other MR methods. Future work will focus on optimizing the use of these multiple 
imaging modalities in concert to plan surgeries in patients with complex partial seizures. 
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ABSTRACT 

We have developed a portable DICOM-compliant low-cost PC NT-based display workstation with 
Asynchronous Transfer Mode (ATM) and Ethernet connectivity to Picture Archiving and 
Communication System (PACS). 

This paper describes the hardware components and software system of the workstation, and 
compares the performance of ATM and Ethernet connectivity in the workstation. The display 
workstation consists of a DOME Md2/PCI board, two 1,600 x 1,280 resolution monitors, and an 
asynchronous transfer mode (ATM) OC-3 communication board. Implemented software includes: 
(a) a DICOM server, which automatically receives images from the DICOM-compliant PACS 
Controller; (b) a patient folder manager, which incorporates the local database to group multiple 
studies from individual patients; (c) a sophisticated display program with automatic image pre- 
processing functions, such as pre-set window and level; and (d) a DICOM query and retrieve 
services to retrieve images from DICOM-compliant archive. 

The display workstation demonstrates that: (1) the PC/NT-based display workstation is 100% 
DICOM-compliant. (2) DICOM query and retrieve services support convenient multi-level 
retrieval of historical images from PACS archive. (3) Integration of DICOM and ATM into the 
display workstation facilities and speeds up image communication. (4) Patient folder management 
provides hierarchical image storage for individual patients, which consequently facilities review of 
image. 

This paper demonstrates that the PC/NT based full-functioned display workstation can be 
implemented cost-effectively for PACS applications. 

Key Words: Picture Archiving and Communication System (PACS), Digital Imaging 
Communication in Medicine (DICOM), Asynchronous Transfer Mode (ATM), display workstation, 
query, retrieve, and personal computer (PC). 



1. Introduction 

The basic and prevalent meaning of PACS is acquisition, storage, manage, display of digital radiological images 
through communication networks, and the integration of PACS with information systems like the hospital 
information system (HIS) and radiological information system (RIS) [1, 2, 3]. At present, more and more 
hospitals are implementing PACS to improve the quality of health care, save doctors’ time for reviewing 
radiological images and reduce the cost of radiology services. Here, inexpensive display workstations play an 
important role for these purposes. It allows radiologists and referring physicians, Emergency Room (ER), 
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Intensive Care Unit (ICU) and Cardiac Care Unit (CCU) staff to access images quickly, directly, conveniently, 
and cost effectively. 

In recent years, the Pentium Personal Computer (PC) combined with Windows NT operating system becomes an 
attractive candidate in PACS workstation design for its high power and reasonable price [4, 5], 

We developed a PC/NT based PACS display workstation with ATM and DICOM Connectivity. The DCIOM 
provides a standard communication protocol and the ATM allows quick image delivery. In this paper, we first 
describe the workstation hardware configuration and software architecture, followed by the performance of 
ATM versus the Ethernet, and the benefit of using the PC/NT based workstation. 

2. Hardware Configuration 

We designed the PC/NT based workstation with the goals of high quality display, fast image communication, 
User friendliness, and cost-effectiveness. The hardware configuration of the workstation is shown in Figure 1. 




Fisure 1 Configuration of the display workstation 



2.1 Host Computer 

The host computer of the workstation is a PC with Intel Pentium Pro 180 MHz, 128 Mbytes RAM, PCI bus 
structure and two 40 Mbps Ultra SCSI disks as local storage. 
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2.2 Display Devices 

Display devices include single or multiple display monitors and a video board. The display monitor is a 24-inch 
1600 x 1280 resolution in portrait mode (Image Systems Corp., Hopkins, MN). A DOME Md2/PCI host adapter 
(DOME Image Systems, Inc., Waltham, MA) as the video board. 

2.3 Networking Equipment 

In the workstation, asynchronous transfer mode (ATM) technology is used as the primary means of 
communication, and Ethernet as the back-up. In the former, we use the FORE 155-Mbps (OC3) ATM host 
adapter and FORE ForeRunner ASX-200BX ATM switch (FORE Systems, Inc. Warrendale, PA). The 3Com 
lOBase-T Etherlink adapter is used for the Ethernet. 



3. Software System 

The software is developed based on Microsoft Windows NT platform and in Visual C/C++ programming 
environment. WinSock communication over TCP/IP, Microsoft Foundation Class (MFC) libraries, DOME 
Image Processing Library (DIMPL), UC Davis DICOM library, and our Windows NT-based PACS API 
libraries are used as development tools. The user interface of the display workstation is icon/menu driven with 
user-friendly graphic. 

3.1 Software Architecture 

The architecture of the software system is divided into four layers: application interface layer, application 
libraries layer, system libraries layer, and OS(operating system) driver layer which is over the hardware layer. 
Figure 2 shows the software architecture of the display workstation. 

The application interface layer is the top layer of the software system that interfaces with the end user of the 
display workstation. This layer is composed of four modules: (1) image communication software package, (2) 
patient folder management, (3) image display program, and (4) DICOM query and retrieve software package. 
This layer directly supports any application that requires accessing PACS and radiological images. 

In the application library layer, we develop the PACS API Libraries which provides all library functions to 
support the four modules in the application interface layer. Here, UC Davis DICOM Network Transport 
Libraries and DIMSE-C Libraries ensure DICOM communication protocols and functions, and DOME Image 
Processing Library (DIMPL) supplies library functions for image display of the workstation. 

The system library layer is responsible for providing WindowsNT system libraries, Win 32 API functions and 
Microsoft Foundation Class (MFC) to serve as a developmental platform. 

The OS driver layer provides Windows NT Operating System and its drivers for connecting with hardware 
components which includes DOME Md2/PCI driver for DOME Md2/PCI video board, ForeRunner 200E driver 
for FORE 155 Mbps ATM adapter and lOBased-T Ethernet driver for lOBase-T Ethernet adapter. 

All data flow between the layers of the software is shown in Figure 3. 
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Figure 2 Software Architecture of the Display Workstation 
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Figure 3 Data flow in the Display Workstation. 



3.2 Software Modules in the Application Interface Layer 

In the workstation, the user has access only at the application interface layer, which is composed of four 
modules described below. 

3.2.1 Image Communication 

The module is responsible for supporting DICOM services with DICOM communication protocols over TCP/IP 
to perform two DICOM services: Storage Service Class Provider (SCP) and Storage Service Class User (SCU). 
The DICOM services include C-Echo for verification, C-Store for storage, C-Find for Querying and C-Move for 
retrieving [6]. 

3.2.2 Patient Folder Management 

This module manages the local storage with hierarchical, or tree-structure, directories to organize patient folders 
within the display workstation. 

DICOM decoder is used to extract patient demographic data and examination records from the header of a 
DICOM image. The reformatter of the module changes the image from DICOM format to DOME format for 



36 













display. The extracted data via the DICOM decoder, and the reformatted image by the reformatter are inserted 
into an individual patient folder. A patient folder contains three hierarchical levels: patient level, study level, and 
series level. The hierarchy starts with a root directory in the local storage system, i.e., hard disks of the display 
workstation. Figure 4 is a diagram of the patient folder structure. 



Root 



Patient #1‘ 



Patient #k~ 



Study #1 



Study #h 



Patient #2 ... 



Series #1 — 
Series #2 ... 



Series #m 



Study #2 ... 



I — Series#! 



Study #1 ... 
Study #2 ... 



Series #2 ... 
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#l:Image, Data 

- #2:Image, Data 

1 — #n:tmage, Data 

' #l:Image, Data 

- #2:Image, Data 
1 — #n:Image, Data 

#l:Image, Data 
‘ #2:Image, Data 
' #n:image, Data 

#l:Image, Data 
#2:Image, Data 
#n:Image, Data 



"Study #h ... 

Figure 4 Three-level hierarchy of the patient folders managed by the display software. 



A patient folder is automatically created in the workstation upon receipt of the first image of the patient. 
Subsequent images from individual studies and series are inserted into the patient folder accordingly. The 
patient folder can be automatically deleted from the workstation based on certain aging criteria such as number 
of days the folder is created, or discharge or transfer of the patient. Figure 5 presents the interface of three 
hierarchical levels of patient folders. 

3.2.3 Image Display Program 

The image display program supports both single and dual 24-inch 1,600 x 1,280-resolution portrait monitors to 
display patient information and radiological images in the workstation. Images with the DOME image header 
format in a patient folder can be displayed via the image display program. 
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Figure 5 Patient folders in the Display Workstation. Each folder contains three hierarchical levels: Patient 
Level (Top), Study Level (Middle), and Series Level (Bottom). 



3.2.3.1 Display Format 

The screen layout of the workstation is user adjustable with one image on one monitor, two on one, four on one, 
etc. In Figure 6, two set of MR images are shown. The program supports multi-modality display for CT, MR 
and CR. Multi-modality images can be mixed with each modality images displayed in a monitor as shown in 
Figure 7 (CR and MR). 

3.2.3.2 Image Manipulation and Processing 

Image manipulation functions such as zoom, pan, rotation, flip, window and level adjustment, and invert are 
available. Automatic defaulted window and level preset function is used during imaging loading to minimize the 
manipulation time. Real-time zoom and contrast adjustment can be easily done by using the mouse. 

3.2.4 Query and Retrieve 

This module is a DICOM query/retrieve service class user (Q/R SCU) to query and retrieve patient studies from 
the PACS long-term archive or directly from radiological imaging systems. The query and retrieve module 
supports DICOM C-Echo, C-store, C-Find, and C-Move services. With this module, the workstation has access 
capability to Query/Retrieve Service Class Providers, which use the Q/R information models of patient root and 
study root. The supported Query/Retrieve levels are shown in Table 1. 
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Figure 6 

Display coronal (left monitor) and axial (right monitor) MR images in the workstation. 
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Figure 7 

Multi-Modality Display in the workstation: Left monitor, CR image; Right monitor, MR images. 
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Table 1 

Supported Query/Retrieve 
Levels in the display 
workstation 
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4. ATM and Ethernet Performance 

Network communication performance plays an important role in the display workstation for quick access of 
radiological images and patient information from the PACS Controller. We have evaluated the ATM and the 
Ethernet performance of the display workstation. 

4.1 Testing Equipment 

Two nodes are used to test the ATM and the Ethernet performance. Each node is equipped with a host computer 
with Pentium Pro 1 80 MHz Processor, 64-Mbyte RAM, Windows NT operating system and ATM and Ethernet 
adapters. Networking Equipment is the Fore PCA-200EPC ATM PCI Host Adapter, Fore ForeRunner ASX- 
200BX ATM Switch and 3Com EtherLink Ethernet Host Adapter (see Section 2). 

4.2 Types of Data Transmission 

We use WinSock communication over TCP/IP to evaluate the ATM and the Ethernet performance of memory- 
to-memory and disk-to-memory transmission rate. We also measure the disk-to-disk transmission rate using the 
FTP. 

4.3 Testing Results 

Figure 8 shows the evaluation result, which demonstrates that ATM can provide much better communication 
performance in the display workstation than Ethernet. For memory-to-memory, ATM is an order of magnitude 
faster. 




0 20 40 60 80 100 



Figure 8 the test results of the A TM and the Ethernet performances 
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5. Conclusion and Discussion 



The PC/Windows NT-based full-functioned diagnostic workstation developed in our laboratory demonstrates 
that it can be implemented cost-effectively for PACS applications. The display workstation provides the 
following capability: 

(1) It is 100% DICOM-compliant. 

(2) DICOM Query and Retrieve services support convenient multi-level retrieval of historical images from 
PACS archive. 

(3) The integration of DICOM communication and ATM network into the display workstation facilitates 
and speeds up the communication of images. 

(4) Automatic window and level preset function minimizes requirement for image manipulation thereby 
improving radiologists’ efficiency. 

(5) Patient folder management provides hierarchical image storage for individual patients, which facilitates 
image review at the display workstation. 

(6) PC/Windows NT-based display workstation costs less than the Unix-based workstation. 

With the fast-developed networking technology for the PCI bus architecture, for examples, 600 Mbps ATM and 
100 Mbps fast Ethernet and gigabit Ethernet, will result in a better image transmission performance of the 
PC/NT-based workstation. 
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ABSTRACT 

We are in the process of conducting a research of full-field direct digital telemammography using three protocols: 
telediagnosis, teleconsultation, and telemanagement. To conduct this research project, an asynchronous transfer mode network 
based telemammography system was developed across two remote campuses in our facility. The hardware and software 
components of this system are detailed. The system was embedded in a clinical environment for a four-month test. Some 
preliminary study results from the current phase of this study are reported. 

Key words: digital mammography, high speed networking, high resolution display, teleradiology 



I. Introduction 

The early detection of breast cancer increases the survival rate in women [1], Today, film-screen mammography is 
the most common and effective technique for the detection of breast cancer [2-4]. However, the film-screen image recording 
system of current mammography has several technical limitations that can reduce the breast cancer diagnostic accuracy. A 
state-of-the-art technology, full-field direct digital mammography (FFDDM), has the potential to increase the sensitivity of 
clinical and screening examinations and emerge as a film screen mammography replacement [5]. It is believed that efficiently 
delivering the superior image quality of FFDDM to expert mammographers will significantly increase the breast cancer 
diagnostic accuracy. This concept motivates the studies of full-field direct digital (FFDD) telemammography. 

In our previous work, we reported on three protocols, telediagnosis, teleconsultation, and telemanagement, to 
evaluate the FFDD telemammography concept [6,7]. We also presented a plan on how to develop such a test-bed for this 
evaluation study. Since then we have implemented the FFDD telemammography system between two campuses: UCSF 
medical center (or main campus) and Mount Zion Hospital (MZH) where our mammography expert center is. These two 
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campuses are two miles apart. In this paper, we first present our newly established asynchronous transfer mode (ATM) 
network-based telemammography system. Then, a four-month clinical test is described. The test results regarding the time 
measurements of the system processing and preliminary image quality studies are reported. Bases on these measured results, 
we estimated the elapsed time required for users to view the FFDDM images either at MZH or at the UCSF main campus. 
Finally, we discuss the possible approaches to optimize the image delivery performance. 



II. Telemammography Test-Bed 

We present the telemammography test-bed in two aspects: hardware configuration and software implementation. In 
the hardware configuration aspect, the major computer and network systems and their associated functions are described. Based 
upon the hardware setup, software programs that we have developed to integrate all the components into a functional 
telemammography system are given. 

II. 1 Hardware configuration 

Currently, the major components in our telemammography system include one FFDDM system (the second system 
will be installed in September 1998), one data management system, two high resolution (2K x 2.5K) digital mammogram 
display workstations, and one Laser film imager. Figure 1 illustrates the configuration of this telemammography system. 

11.1.1 FFDDM system 

The FFDDM system serves as the image acquisition component in the telemammography system. It uses a slot- 
scanning based technology with a novel Charge-Coupled Device (CCD) detector from Fischer Imaging Corporation. (Denver, 
CO) [8,9], The system contains an imaging unit and an acquisition computer which is a Sun Ultra-I computer system 
(Mountain View, CA). The imaging unit and the acquisition computer are linked by an electronic board developed by 
Fischer. 

The image area of the FFDDM system is a rectangular of 240 x 300 mm 2 . The spatial resolution of a generated 
image is 54 x 54 pm 2 per pixel in nominal and 27 x 27 pm 2 in high resolution which is approximately to 9 line-pairs (lp) 
per mm and 18 lp/mm, respectively. The matrix size of the image is 4,096 x 5,625 (4K x 5.5K) pixels, with 12 bits/pixel 
which is stored within 2 bytes. As a result, each image is about 46 Mbytes. 

11. 1.2 Data management system 

The data management system functions as the database server for image archival and retrieval. The system is 
composed of a Sun Enterprise-3000 computer, a 42-Gbyte Sun Network Array Disk, and five fast-wide small computer 
standard interface (SCSI) disks (4 x 4.2 Gbytes and 1 x 2.1 Gbytes). In this management system, we store the digital 
mammogram data either in the Array Disk of in the SCSI disks while patient demographic information is managed by an 
Oracle database software. 
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11. 1.3 Digital mammogram display workstations 

The two display workstations provide soft copy display for this telemammography system. Each of the display 
workstations consist of the following components: 

(1) one Sun Ultra-II that contains one 200-MHz CPU, 512-MByte CPU memory, and a 4.2-Gbyte fast-wide SCSI 
disk, 

(2) two DOME MD5-SBX display boards; each is assembled by 5 mega-pixel video RAM, one 10-bit digital to 
analog converter, and one 16-bit to 8-bit look-up-table, and 

(3) two Data Ray monitors (Westminster, CO); each provides 2,048 x 2,560 pixel matrix, 64 Hz frame rate, and 120 
foot-lamber (maximum light output). 

The image display and manipulation program is developed in-house and based on X-Motif and DOME image 
processing software libraries. 

11.1.4 Laser film imager 

The laser film imager prints hard copy (films) which are used to compare image quality between soft copy display 
and hard copy film. Specifically speaking, the film imager includes two components: a PC computer and the laser imager. 
The PC computer serves as an interface between other computer systems and the laser imager via an Ethernet. The laser 
imager is able to photograph 4,096 x 5,120 pixels onto a film with 4096 gray levels. 

11.1.5 ATM network 

All the components stated above (except for the laser film imager) are linked with an ATM network. In this ATM 
network configuration, two switches (one at each of the two campus) communicate through a UCSF Sonet ring sharing an 
OC-3C bandwidth (155 Mbits/second). Locally, the switches serves as a root to exchange information among the computer 
systems. 

II.2 Software implementation 

In the present stage of the system development, our aim is to assure the images smoothly flow from the FFDDM 
system to the display workstations. The fundamental software includes DICOM communication programs, a workstation 
database program, and an image display and manipulation program. 

II.2.1 DICOM communication software 

We use the Digital Imaging and Communications in Medicine (DICOM) communication protocols to transfer 
images such as storage, query, retrieve, and print service classes. The images generated by the FFDDM system contain the 
file format and information following DICOM information definitions. Digital mammograms transferred between computer 
systems are based on the DICOM storage service class. Currently, the data management computer system and the two display 
workstations can provide roles of service class provider (SCP) and service class user (SCU). The FFDDM acquisition 
computer only acts as an image client. The DICOM query/retrieve service class is established between the data management 
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computer only acts as an image client. The DICOM query/retrieve service class is established between the data management 
system and the two display workstations. The PC computer associated with the laser film imager provides the DICOM print 
class service. In the current stage, we only configure this service class between the PC and the digital mammogram display 
workstation in MZH. 

11.2.2 Workstation database software 

Before a digital mammogram is ready for users to display in the image display workstation, it is processed by a 
workstation database software. The process includes (1) dividing the DICOM format file into a DICOM header file and an 
image pixel file, and (2) extracting information from the header file to form a local database. The database is a three-layer 
hierarchy which keeps track of information about patients, examinations, and images. The database allows us to present a 
patient list GUI in the image display software as described in Section 13.2.3. In addition, it provides a means to remove old 
images from the local storage to maintain enough space for new images. Attributes such as patient name, hospital 
identification number, and birth date are recorded for locating a particular patient. It is possible for patients to have multiple 
mammography examinations. The patient key is designed to use a combination of the examination date, time, and patient 
hospital identification number to guarantee that each examination is unique Each examination may contain several images. 
A description of the patient positioning, such as breast laterality and view, is saved in the database. 

11.2.3 Image display and manipulation software 

A window-based image display and manipulation software is developed for providing users with a GUI environment 
for accessing patient information including text and images. A patient list allows users to browse through patients and their 
associated examinations and images. Images may be displayed in 1:1 (one image per monitor), 2:1, or 4:1 format using the 
two 2K display monitors. Since the image matrix is 4,096 x 5,625, none of the above display modes can present a full 
resolution FFDDM image on a single monitor. It is possible the portion of the image displayed on the monitors is 
background air. To insure the relevant image information is displayed, the breast laterality is used to set the output rectangle 
of the image. The images in any of the display modes can be instantly expanded to full resolution (4K x 5.6K) in 1:1 format, 
and vice versa. With these two image display features unnecessary operations of the display workstation are minimized. In 
addition, functions such as pan-and-scroll, window-and-level, scalable magnifying glass, rotation, and on-screen image 
exchange are provided to facilitate users' viewing practice. 



III. Clinical Test and Preliminary Results 



III.l Patient collection 

The telemammography system has been released for clinical testing since September 1997. Based on our clinical 
protocol, patients who have completed a film-screen mammography may have two additional FFDD mammograms of one 
breast with informed consent. This is conducted as follows: 
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(1) Patient examination involves normal and abnormal cases. 

(2) Normal patients can be recruited from regular screening examinations. 

(3) The abnormal case selection is conducted by the chief of the Breast Imaging Section during conventional 
mammography readings. From the reading, those patients suspected to have masses or calcifications were called 
in for FFDDM examinations. 

(4) Only one cranio-caudal (CC) view and one medio-lateral oblique (MLO) view would be taken. 

In four months, we have examined thirty-one patients with the FFDDM system. 

111.2 Telemammography operations 

Digital mammograms acquired by the FFDDM system are first sub-sampled to 600 x 800 pixels in the acquisition 
computer memory and displayed on the console monitor for quality assurance. The technician may adjust the contrast and 
brightness of the image to assess its quality. If the image quality is poor, it may be rejected and another scan performed. If 
the image quality is good, the mammogram is automatically transmitted to the display workstation at MZH through the 
ATM network using the DICOM storage class service. Once the image is received at the workstation, it is first processed by 
the workstation database program; then it is queued and transmitted to the data management system located at the UCSF 
campus through the Sonet Ring. The data management system then relays the image to the second image display 
workstation at the main campus. 

We embedded logging software to show the elapsed time spent in the image delivery and display programs. Our 
preliminary results indicate the image transfer performance of the local area ATM is better than that of wide area ATM. It 
takes 5-6 seconds to transfer one 46-MByte mammogram from one computer system to another (disk to disk) in ATM 
network. The image display workstation database program takes about 25 seconds to process one image. For image display, 
we measured that it requires about 8-9 seconds to display one digital mammogram from the local fast-wide SCSI onto the 
image display monitor. 

111.3 Preliminary image quality study 

All images were viewed using the digital mammography display workstation. Preliminary results indicate that the 
FFDDM images displayed on the display workstation are of good quality. All the lesions shown on conventional 
mammograms can be clearly identified on the workstation with add-on values of enhancement and manipulation features. 
Figures 2 shows an example case that soft copy display is particularly useful for viewing dense breast tissues within which 
mammographers often have difficulty in detecting lesions. Both monitors show the same FFDDM image with different 
window and level settings. In the left monitor, the window and level values present generally good contrast and brightness for 
the entire image. However, the region of dense breast tissues enlarged by the magnifying glass is completely washed out. 
The image can be manipulated by applying a different set of window and level parameters as shown in the right monitor. 
Here, more detailed structure information can be perceived in the region of dense tissues although the image as a whole seems 
to be dim. We have also printed two patient cases with the Konica laser imager for comparison of image quality. The printed 
image quality is comparable to that of the conventional mammogram. 
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IV. Discussions 



The elapsed time from the moment when a digital mammogram is accepted by a technologist in the FFDDM system 
to the time when the image is displayed on the MZH display workstation includes (1) the transmission time from the 
FFDDM acquisition computer to the workstation, (2) the queue time and the workstation database processing time, and (3) 
the image display time. If the image is also required to be displayed on the UCSF campus workstation, the elapsed time also 
includes: (1) the queue time and the transmission time from the MZH workstation to the data management system, (2) the 
queue time and transmission time from the data management system to the UCSF campus workstation, (3) the queue time 
and the workstation database processing time, and (4) the image display time. The queue time for each of the processes is set 
to 10 seconds. Based on the image transfer and display performances described in Section m.2, it is estimated that the elapsed 
time for an image to be displayed in the MZH and UCSF campus display workstations is about 50 seconds and 120 seconds, 
respectively. 

The telemammography operations described in Section m.2 were for testing the entire image flow of the system. It 
is possible to independently transfer images from the FFDDM acquisition computer to the image display workstations in 
both at MZH and at the main campus. In this way, the time duration spent in the data management system and associated 
process queue time can be eliminated. As a result, it will be faster for users to view the image at the UCSF campus 
workstation. On the other hand, the workload of the FFDDM system will be increased which may slow down the following 
imaging procedures for the technologist. 



V. Future Work 

We have completed the first phase of a three year comprehensive program in developing the application of 
telemammography. This phase is to resolve the technical issues of the telemammography system as discussed in this paper. 
The second phase is to set up a second FFDDM system in the UCSF Ambulatory Care Center where the mission of health 
care delivery is toward out patient. The goal is to compare the mammography service between the two sites with different 
operating environment, one being the expert site while the other is an outpatient clinic. The last phase is to collect sufficient 
patient data to evaluate the effectiveness and efficacy of telemammography with the three clinical protocols. 
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ABSTRACT 

Real-time consultation between referring physicians or radiologists with an expert is 
critical for timely and adequate management of problem cases. During consultation, both sides 
need to (1) synchronously manipulate high resolution digital radiographic images or large volume 
MR/CT images, (2) perform interpretation interactively, and (3) converse with audio. We present 
a specific designed teleconsultation system with bi-directional remote control technology to meet 
critical teleconsultation application with high resolution and large volume medical images in a 
limited bandwidth network environment. We give the system design and implementation 
methods, and also describe the teleconsultation procedure and protocol used in this system. 
Finally, preliminary results are discussed. 



Keywords: Teleconsultation, medical imaging, bi-directional remote control, networked multi- 
media, teleconsultation protocol, collaborative health care. 

1. INTRODUCTION 

The managed care trend in health care delivery system expedites the formation of 
teleradiology expert centers. In the expert center model of teleradiology [1], the medical images 
are sent from the examination sites to a remote site where an expert radiologist will make the 
diagnosis. The report is sent to the examination site where a primary physician can then prescribe 
the patient’s treatment immediately. During teleradiology, real-time consultation between general 
physicians or radiologists with an expert to study the same images interactively are sometimes 
required. In real-time consultation, both sides need to synchronously manipulate and interpret 
high-resolution digital radiographic images (16 MB/exam.) and/or large volume MR/CT 
sequential images (8-20 MB/exam.). Here, we present a specific designed teleconsultation system 
with bi-directional remote control technology to meet critical teleconsultation application with 
high-resolution and large-volume medical images in a limited bandwidth network environment. 

2. BACKGROUND AND REQUIREMENTS 

The University of California at San Francisco (UCSF) comprises of two medical centers. 
The distance between the main campus (UCSF Medical Center) and Mt. Zion hospital (MZH) is 
2 km. There are almost 1.0 gigabytes/day of MR/CT images transmitted through an asynchronous 
transfer mode (ATM) network (T1 line as backup) from MZH to UCSF Medical Center. These 
images are first sent to the clinical PACS server and then routed to display workstations [2] for 
expert to make diagnosis. Usually, referring physicians or radiologists view films at MZH, and 
when necessary, would request die expert to view softcopy images in display workstations at 
UCSF. They would discuss the cases through the telephone. With this type of consultation, both 
parties can not synchronize their. operation on the images and can not manipulation them (e.g., to 
use cine mode, zoom and window/level operations) collaboratively. 
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In order to provide real-time consultation services for serious or difficult cases, a real-time 
teleconsultation system has been developed in our laboratory which is based on a distributed data 
storage in a shared window environment. This teleconsultation system meets the following 
requirements: 

1) Provide real-time teleconsultation services with high-resolution and large volume 
medical images (MR/CT, CR, US, Mammography); 

2) Synchronously manipulate images on both local and remote sites including remote 
cursor, window/level, zoom, cine mode, overlay, and measurement; 

3) Support multimedia including audio and video (option) communications; 

4) Interface to PACS through the DICOM standard; 

5) Can be used in intranet (LAN) and internet (WAN) environment with TCP/IP 
protocol; 

6) Scalable network connections including ATM, Ethernet, and modem; 

7) Use low cost NT/PC based system. 



3. SYSTEM DESIGN AND IMPLEMENTATION 

In teleconsultation, it involves a referring site and an expert site. Between these sites three 
kinds of data communication are necessary: images transferred from the referring physician or 
general radiologist site to the expert site; remote control messages between the two sites; and 
voice communication between the referring physician and the expert. Among these three types of 
data, transmitting images requires very high bandwidth network but they normally can be 
preloaded before the consultation session unless it is an emergency case. Images preloading can 
be done with various kinds of network, e. g., ATM, T1 line, Ethernet, or telephone line through 
non-peak hour, evenings or weekends dependent on the turn around time requirement. Remote 
control message routing and voice communication are real-time on-line procedures but with low 
bandwidth requirement. 



General physician or 
radiologist site 



Expert site 



Telephone 







Tele-consultation 

workstation 



Network 







Tele-consultation 

workstation 



Telephone 



Telephone line 



Figure 1 . Diagram of teleconsultation system 

Based on the above analysis and our PACS network infrastructure [3], we have designed a 
real-time teleconsultation system and implemented it at UCSF. Figure 1 . shows the diagram of 
the teleconsultation system. 
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According to the expert model of teleradiology, the teleconsultation system is composed of 
two sites: the expert and the general physician or radiologist site. The hardware configuration of 
the teleconsultation system at each site includes: 

1) One Pentium II NT workstations, each with 128 MB SDRAM, and 4 GB hard disk, 
as well as Ethernet, ATM and modem connections. One workstation located at the 
expert site and the other at the general physician site ; 

2) One DOME Md2/PCI high resolution display boards which can be configured to 
support single or dual monitors; 

3) Two high resolution gray scale monitors (Image System Inc.); 

4) One telephones for audio communication. 

The software modules (TeleConApp) in the teleconsultation system consists of: 

1) Image display graphic user interface for teleconsultation; 

2) Event Interpreter for local and remote message dispatching; 

3) Remote Control Manager for routing messages between local and remote sites; 

4) View and Memory Manager for view controlling and image data memory 
management; 

5) Graphic Manager for image rendering and display; 

6) DICOM communication services for DICOM image receiving and sending between 
the teleconsultation system, scanners and PACS server; 

7) Image database for medical image display and teleconsultation authoring. 



View station 




(b) (c) 

Figure 2. Network connection of teleconsultation system in clinical environment, (a) general 
physician or radiologist site; (b) PACS central archiving; (c) expert site. 
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The data flow and network connection of teleconsultation system in clinic network 
environment is shown in Figure 2. There are three parts in Figure 2: (a) The General physician or 
radiologist site; (b) PACS central archiving; and (c) The expert site. 

4. TELECONSULTATION PROCEDURE AND PROTOCOL 

A teleconsultation session proceeds as follows. When cases are needed to be consulted, a 
general physician or radiologist first collects the images from the scanners, display workstations 
or PACS image database, and pushes them to the teleconsultation workstation located at the local 
reading room. At the teleconsultation workstation, the referring physician then performs data 
authoring supplied by the teleconsultation system and sends them to the teleconsultation 
workstation located at the expert site through either the local area network if it is within the same 
building or campus, or through the wide area network if it is distant away. Later, either the 
general physician or an expert can call the other site to start the consultation session. 

We develop a teleconsultation protocol with three steps: data formatting; data authoring 
and data presentation as follows: 

Data formatting : The first step is to convert all input images for teleconsultation to the 
DICOM standard. This is done at the acquisition gateway or the PACS server. 

Data authoring'. The authoring procedure is to authorize data for teleconsultation. The 
authoring function module is an integrated component in the software package “TeleConApp”. 
Figure 3 shows the data authoring procedure. There are three steps in data authoring: 
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Figure 3. Data authoring steps 

1) The general physician or radiologist uses the TeleConApp program to create an object 
called the virtual envelope which includes information of the selected patients, studies, 
and series; as well as the host name of the expert site, the name of the consultant from 
the teleconsultation local database. Note that the virtue envelope does not contain 
images. The virtual envelope is sent to the expert site through a network with DICOM 
communication services; 

2) After the expert site receives the envelope, either site can create a consultation session 
and the general physician site sends the DICOM image objects related to the patient as 
dictated by the virtual envelope to the expert site; 

3) After sending images, the general physician site automatically performs a DICOM 
query to the expert site, and the expert site verifies the receipt of the virtual envelope 
and image object to the general physician site. The consultation session can start once 
the data is verified. 

Data presentation : Before the data presentation, both the expert and the general physician 
site have the virtual envelope and image data. Either site can operate the TeleConApp program to 
display and manipulate images and related information. TeleConApp synchronizes the operation 
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of image display and manipulation at both sites. Figure 4 shows the teleconsultation operation 
procedure in either site: 

1) Either the expert or general physician site can start the consultation procedure by 
selecting a session and load image objects from the site's local database to the memory 
and display them at both sites (5,6). 

2) Either site can manipulate the displayed images at the User Interface Window to create 
events, which are received by Event Interpreter. The site starts an event is called the 
local, whereas the other is the remote. 

3) Event Interpreter sends the events to the local View Manager and Remote Control 
Manager. 

4) Remote Control Manager encodes events and sends them to the remote site (Note: If the 
general physician site starts the event, it is the local; whereas the expert is the remote. 
On the other hand, if the expert site starts the event, it is the local and the physician site is 
the remote.). At the remote site, events are received and decoded by Remote Control 
Manager and send to View Manager ( shaded (4) and dotted line). 

5) View Manager handles both local and remote events and processes images according to 
the event types and sends processed image data to Graphic Manager; 

6) After receiving processed image data, Graphic Manager renders them to display 
windows, which are managed by User Interface Window. 



DICOM 

◄ ► 




Figure 4. Operation procedure of data presentation in the local site during a teleconsultation 
session. Shaded area and dotted line represent events happened at the remote site. 
Numerals represent the data flow. 
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5. PRELIMINARY RESULTS 

We selected 20 patients with more than 2,000 different modality images including GE 
MR/CT, Siemens CT, Fuji and Agfa CR, and pushed them from PACS server to one 
teleconsultation workstation through PACS network with the DICOM communication standard. 
Then we authored the data in the workstation and created teleconsultation sessions in both sites. 

We performed the teleconsultation operations through intranet and internet with Ethernet 
and modem connections, respectively. During consultation, both sites manipulated images, did 
ROI and talked with telephones. There were no crash, no data lost, no delay feeling on both sites 
in a two month testing in the laboratory setting. The system was exhibited at InfoRAD, RSNA 97, 
Chicago, IL [4]. 

Currently, we are in the process of deploying the teleconsultation system to Neuro-imaging 
Section at UCSF and MZH for clinical evaluation. 



6. CONCLUSIONS 

We have developed a cost-effective teleconsultation system for collaborative health care 
application. This system provides data authoring, and synchronizes image display and 
manipulation on both the general physician/radiologist and the expert site during consultation 
through remote cursors. This system allows real-time collaborative consultation of serious or 
difficult cases with high resolution and large volume medical images in a limited bandwidth 
network environment. 
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ABSTRACT 

DICOM provides integration solutions for PACS to interconnect multi-vendor imaging equipment, 
regardless of computer hardware and operating system platforms, for communication of images. This paper describes 
our three-year experience integrating heterogeneous imaging systems into a large-scale PACS based on DICOM. 

Vendor imaging equipment connected to the PACS include (a) six CT and MR imaging devices; (b) two 
CR systems; (c) two laser film digitizers; (d) one commercial ultrasound PACS module; and (e) eight DICOM- 
based display workstations. The image communication software implemented in the PACS was modified from 
Mallinckrodt's Central Test Node software and University of California at Davis' DICOM software to support the 
Unix-based and Windows NT-based computer systems, respectively. Images acquired from the individual CT/MR 
devices, CR systems, film digitizers, and the commercial PACS module were transmitted from the acquisition 
computers to the PACS Controller, where these images were routed to the display workstations and the archive 
server. Problems during integration included: (a) missing slice image(s) from an image sequence during the 
acquisition process; (b) incorrect data encoding in image header; (c) DICOM Service Object Pair Class not 
supported by the Service Class Provider vendor; and (d) DICOM conformance mismatching and shadow group 
conflict between the individual vendors. These problems were resolved by the mechanisms we implemented in our 
Image Acquisition Gateways and PACS Controller. 

The DICOM communications standard provides a simple means of interconnecting multi-vendor equipment 
for communication of images. However, operating a large-scale PACS configured with heterogeneous imaging 
systems requires significant effort in software integration to ensure data integrity and operation efficiency. 

Keywords: Digital imaging and communications in medicine (DICOM), image communication, service object 
pair (SOP) service class, service class provider (SCP), service class user (SCU). 



1. INTRODUCTION 



The four major components of a Picture Archiving and Communication System (PACS) are; 
acquisition, archive, display, and communication. Images generated from the radiologic imaging devices 
such as the computed tomographic (CT) systems, magnetic resonance (MR) imagers, and computed 
radiographic (CR) systems are acquired by the acquisition computers. These images are then transmitted 
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to the archive server, where they are routed to the display workstations for manipulation and review, and 
to the storage subsystem for long-term and short-term archiving. 

The Digital Imaging and Communications in Medicine (DICOM) standard is intended to promote a 
generic communication method for heterogeneous imaging systems, allowing the transfer of images and 
associated information. By adapting the DICOM standard, a PACS is capable of interconnecting its 
individual components and the radiologic imaging devices to support image acquisition, archiving, and 
display. However, imaging equipment vendors often take different DICOM compliant implementations 
for their systems, which consequently lead to certain difficulties for these systems to interoperate. 

We have implemented in our PACS two mechanisms that incorporate our DICOM-based 
applications to ensure data integrity and to guarantee interoperability. One mechanism was installed in the 
acquisition computers to provide a reliable and efficient process in acquiring images from the radiologic 
imaging devices; and the other was employed in the archive server to provide interoperability for multi- 
vendor imaging systems in the transfer of images. 

This paper describes our three-year experience integrating multi- vendor imaging equipment into a 
large-scale PACS, and the data integrity and interoperability problems we encountered during the 
DICOM/PACS integration. 



2. MATERIALS AND METHODS 



2.1 DICOM Communication 

Two major DICOM Service Object Pair (SOP) Classes used in our PACS for communication of 
images are the Storage Service Class and the Query/Retrieve (Q/R) Service Class: 

• Storage Service Class allows a PACS application running on vendor A's system (i.e. a 
CT scanner) to play the role of a Storage Service Class User (SCU) that initiates 
storage requests and transmits images to vendor B's system (i.e. an image acquisition 
computer), which serves as a Storage Service Class Provider (SCP) accepting images 
into its local storage device. 

• Q/R Service Class allows PACS applications running on vendor A's system (i.e. a 
display station) to play the role of a Q/R SCU that queries and retrieves images from 
vendor B' system (i.e. an archive server), which serves as a Q/R SCP processing 
query and retrieval requests. 

Figure 1 illustrates the communication of images utilizing the Storage Service Class and Q/R Service Class 
in a PACS environment. 

The communication software used in our PACS are based on Mallinckrodt's Central Test Node 
(CTN) software and University of California at Davis’ (UCD) DICOM software, which support our 
Sun/Unix-based and PC/Windows NT-based computer systems, respectively. These two software 
packages were modified to include several enhanced features that provide reliable and efficient 
communication for images. For more detailed information regarding our DICOM communication 
software, please refer to the '97 SPIE Proceedings paper entitled "Adaptation of DICOM to an operational 
PACS." 1 
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Figure 1 . Image communication utilizing DICOM SOP services in PACS. Images generated from the radiologic 
imaging devices are acquired by the image acquisition computers via the Storage SOP service. These images are then 
transmitted to the archive server, where they are routed to the storage subsystem and the display workstations. The archive 
server supports Query/Retrieve (Q/R) SOP service, handling all Q/R requests from the display workstations. 
Abbreviations: SOP - Service Object Pair; SCU - Service Class User; SCP - Service Class Provider. 



2.2 Networking with Multi-Vendor Imaging Equipment 

The communication network implemented for the UCSF PACS is based on a two-tiered 
architecture composed of the OC-3 (155 mbps) Asynchronous Transfer Mode (ATM) and the lOBaseT 
(100 mbps) Ethernet networks 2 ’ 3 ’ 4 . This two-tiered network interconnects four major campus buildings 
(Moffitt and Long Hospitals, UC Imaging Center, and the Campus Library) and a remote affiliated 
hospital (Mt. Zion Hospital; 2 kilometers from UCSF), allowing the communication of images and 
associated information. 

In the past three years, we have successfully integrated the following vendor-specific imaging 
systems with DICOM and connected them to the infrastructure of our PACS : 

(1) six computed tomography (CT) and magnetic resonance (MR) imaging devices; 

(2) two computed radiographic (CR) systems; 

(3) two laser film digitizers; 

(4) one commercial ultrasound (US) PACS module; and 

(5) eight DICOM-based display workstations. 
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By utilizing the DICOM communications standard, images generated from the imaging devices are 
acquired by the image acquisition gateways (IAGs). These images are then transmitted to the PACS 
Controller (Archive Server), where they are routed to the storage subsystem and display workstations. 
Figure 2 illustrates the connections and data flow between the imaging devices, the IAGs, the PACS 
Controller, and the display workstations. 
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Figure 2. Image communication in PACS. Images from the imaging devices are acquired by the Image 
Acquisition Gateways (IAGs) and transmitted to the PACS Controller (Archive Server ) , where they are routed 
to the display workstations (DSs) for manipulation and reviw, and to the storage subsystem for long-term and 
short-term archiving. 



3. RESULTS 



The problems during the integration of the aforementioned imaging systems with DICOM included 
the following: 

(1) Missing slice image(s) from an image sequence during an acquisition process 

When a CT or MR scanner (Storage SCU) transmits individual slice images from an image 
sequence to an IAG (Storage SCP), the scanner initiates the transfer with a a push operation. The 
reliability of transmission is dependent on the transfer mechanism implemented by the scanner. 
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(2) Incorrect data encoding in image header 

Some of these examples are: 

(a) missing Type 1 data elements, which are mandatory attributes in the DICOM standard; 

(b) incorrect value representation (VR) of data elements; and 

(c) encoded data in data elements exceeding their maximum length. 

(3) SOP Service Class not fully supported by the SCP vendor 

This happens when a SCU vendor and a SCP vendor implement a SOP Service Class with 
different operation supports. For example, a C-GET request initiated from a display workstation is 
rejected by an archive server because the latter only accepts C-MOVE requests, even though both 
C-GET and C-MOVE are DICOM standard DIMSE-C operations that support the Q/R Service 
Class. 

(4) DICOM conformance mismatching between the individual vendors 

When a SCU vendor and a SCP vendor implement the Q/R Service Class in different 
Information Models or with different support levels, a C-FIND request in Patient Level initiated 
from a display workstation is rejected by a CT scanner because the latter only supports the Q/R 
Study Root model, which does not accept any query requests in the Patient Level. 

(5) Shadow group conflict between the individual vendors 

When both vendor A and vendor B store their proprietary data in the same shadow group, 
data previously stored in the shadow group by vendor A is overwritten by vendor B's data. 



4. IMAGE ACQUISITION GATEWAYS AND PACS CONTROLLER 



Based on our experience with the PACS/DICOM integration, we have implemented two 
mechanisms incorporating the DICOM-based applications to provide better connectivity solutions for 
multi-vendor imaging equipment in a large-scale PACS environment. 

One of these mechanism is operated in the IAGs to provide a reliable and efficient process for 
acquiring images from the radiologic imaging devices; the other is operated in the PACS Controller to 
provide integration solutions for multi-vendor imaging equipment, allowing their interoperability for the 
transfer of images. The configurations of the IAGs and PACS Controller, and the implemented 
mechanisms are described in the following sections. 
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4.1 Image Acquisition Gateways 
(A) System Configuration 

The I AGs are based on two computer system platforms: the Sun Microsystems computers 
and the Personal Computers (PCs). Each IAG serves as an acquisition node and can be connected 
to one or more radiologic imaging devices for the acquisition of images. 

Sun-Based IAG 



• Solaris operating system 

• C programming language 

• ATM and Ethernet network interfaces 

• Unix socket communication over TCP/IP 

• Mallinckrodt CTN-based DICOM communication software 

• Informix local database 

PC-Based IAG 



• Windows NT operating system 

• C and C++ programming languages 

• ATM and Ethernet network interfaces 

• WinSock communication over TCP/IP 

• UCD DICOM-based communication software 

• Microsoft Access local database 

(B) Software Modules 

The software running on an IAG supports two types of image acquisition, the push-mode 
and the pull-mode operations. 

(a) Push Mode 

Push-mode operation utilizes DICOM’ s Storage SOP service. An imaging device such 
as a CT scanner takes the role of a storage SCU initiating storage requests. The requesting 
IAG (storage SCP) accepts these requests and receives the images. 

(b) Pull Mode 

Pull-mode operation, on the other hand, utilizes DICOM’ s Q/R SOP service. An IAG 
plays the role of a Q/R SCU initiating query requests, selecting desired images, and retrieving 
images from an imaging device (Q/R SCP). Pull-mode operation requires the image 
acquisition process running on an IAG to incorporate IAG’s local database to perform data 
integrity checking. This checking mechanism ensures that no images are lost during the 
acquisition process. 
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Figure 3 summarizes the characteristics of these two operations. 




Figure 3. Interprocess communication among the major processes running on an image acquisition gateway (IAG). 
IAG supports both push and pull operations for acquiring images from the scanners. The DcmPull process 
incorporates LAG'S local database to perform data integrity checking, ensuring no missing slice images from any image 
sequences during the acquisition process. 

The ImgTrack process performs data integrity checking using the following procedures: 

(1) Query study information from the scanners 

(2) Generate acquisition status table 

(3) Periodically check acquisition status of individual image sequences 

(4) Invoke DcmPull process to retrieve images from the scanners 

The DcmPull process, when invoked by the ImgTrack process, will retrieve desired images from 
the scanners and update the acquisition status table accordingly. 

4.2 PACS Controller 

(A) System Configuration 

The PACS Controller, a Sun Microsystems Ultra Enterprise 3000 Multi-CPU server running 
Solaris operating system, is configured with the following components: 

• Redundant array of inexpensive disks (RAID) cache storage 

• Digital video disc (DVD) long-term storage! 

• ATM and Ethernet network interfaces 
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. Mallinckrodt CTN-based DICOM communication software 
• Oracle database 

tThe DVD storage subsystem will be implemented when it becomes available. 

(B) Software Modules 

The PACS Controller supports communication for images with the following DICOM 
implementations: 

(1) a Storage SCP that accepts images from the IAGs; 

(2) a Storage SCU that routes images to the display workstations; and 

(3) a Q/R SCP that handles query and retrieval requests from the display workstations 

Figure 4 illustrates the management of images in the PACS Controller. The DcmBroker process 
serves as an arbitrator ensuring images acquired from the radiologic imaging devices can be 
adequately distributed to the display workstations. This process checks the individual images in the 
PACS Controller's cache storage to determine if any changes in their headers are necessary. The 
major tasks of the DcmBroker process include: 

(1) Correct any mis-coded data in the existing elements; 

(2) modify existing elements; 

(3) add new elements or shadow groups; and 

(4) remove existing elements or shadow groups. 




Figure 4. Interprocess communication among the major processes running on the PACS Controller. The DcmBroker 
serves as an arbitrator ensuring images acquired from the radiologic imaging devices can be adequately distributed to the 
display workstations. 
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5. CONCLUSION 



DICOM provides a simple means of interconnecting multi-vendor imaging equipment for 
communication of images, regardless of computer hardware and operating system platforms. However, 
while DICOM is intended to promote a communications standard towards connectivity, it does not 
necessarily guarantee interoperability. Because of vendors' different approaches in their DICOM 
implementations, one major challenge in PACS integration using DICOM has been the interconnections of 
vendor-specific imaging systems, allowing their mutual communications for images and associated 
information. 

Two major aspects, data integrity and interoperability, are worth considering when integrating 
multi-vendor imaging equipment with DICOM into PACS. Unfortunately, the limitations of access to 
vendor-specific applications have often led to certain difficulties for PACS to interoperate the imaging 
equipment in a multi-vendor environment. As a consequence, interoperation failure of these equipment 
means loss of images and relevant information, which is definitely unacceptable in a clinical setting. 

Our implemented mechanisms to ensure data integrity and cross-platform interoperability have 
significantly improved system reliability and operation efficiency of our PACS. The key issue of these 
improvements is to allow our in-house PACS components such as the IAGs and the PACS Controller to 
take most of the control in the communications with vendor-specific imaging systems. However, 
operating a large-scale PACS configured with multi- vendor imaging equipment requires significant effort 
in software integration in order to achieve a fully operational system. 
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ABBREVIATIONS 



ROI 

PROI 

CROI 

EROI 

LHFC 

HLFC 

LLFC 

HHDC 

PBA 

CBA 

RBA 

SEF 

EM 



region of interest 
phalangeal region of interest 
carpal bones region of interest 
epiphyseal region of interest 

horizontal Low and vertical High Frequency Component 

horizontal High and vertical Low Frequency Component 

horizontal Low and vertical Low Frequency Component 

horizontal High and vertical High Frequency Component 

phalangeal bone age 

carpal bone age 

radiological bone age 

stage of epiphyseal fusion 

energy measure 
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5.1. INTRODUCTION 

Part of the motivation for a computer-assisted diagnosis stems from our desire 
to understand the way in which human intelligence perceives the objects and, 
based on the visual perception, solves the pattern recognition tasks. Visual 
perception, being part of the pattern recognition leads to an extraction of 
certain elements from the object content. These elements, including certain 
parts of the object as well as their color, brightness, density, shape, size, 
etc. and often being very imprecisely defined even by the observer himself, 
serve as a basis for the object recognition. The larger the similarity between 
objects to be compared the many more details are necessary to make a 
distinction. Moreover, the details become more and more sophisticated and 
may be searched by the observer interactively while the recognition process 
proceeds. The interactive perception-recognition process may also proceed 
unconsciously. 

Zadeh 1 has defined a human pattern recognition as a mapping of a 
pattern from pattern space into class-membership space. This kind of 
mapping carried out by nature can be thought of in terms of an opaque 
mapping. Unfortunately, very often neither the recognition process as such is 
understandable nor many details are accessible. In order to be implemented 
as a computer-aided recognition approach, the opaque mapping has to be 
replaced with a transparent mapping described in terms of an algorithm, 
understandable and able to perform on a computer. 

Curiosity, whether we are able to make computers “intelligent” is not the 
only motivation. In clinical application we deal with problems which have 
to be solved in a fast and objective way. However, human observation is 
influenced by internal (coming form the observer) as well as external (often 
independent from the observer) impacts. The objectivity of classification 
is restricted by the receptivity of human senses which are influenced by 
the experiences or level of training, psychological conditions (tiredness, 
hurry, etc.), as well as external conditions (lighting, destructive noise, 
etc.). A failure in perception questions the entire recognition process. 

' The recognition process itself, influenced also by the above mentioned 
conditions, may cause a slow down and/or lead to a false diagnosis. Therefore, 
the diagnostic results are evaluated by the inter-observer (multi-observer 
perfo rmin g one observation) and intra-observer (one observer performing 
multiple observations) discrepancy. The first evaluation reflects more the 
level of training and experiences, whereas, the other one — the replicability 
of the recognition method. Both can assess the influence of the changing 
conditions. 

The implementation of computers in the medical diagnosis improves 
the analysis in three different areas. Firstly, it increases the objectivity by 
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using quantitative features instead of a visual interpolation. These features 
are extracted automatically rather than manually in order to accelerate 
the analysis and make the measurements independent from the external 
conditions (light), measurement inexactness, or errors (calibration errors, 
etc.). Secondly, a transparent algorithm is defined to standardize the decision- 
making process in the feature analysis. This again increases the objectivity 
and solves the problem of replicability. Finally, it permits a set of images 
and features describing them to be collected and recognized as a medical 
standard. 

In this approach we refer to the human observation performed in the clinical 
routine. Therefore, before going into the technical details, let us look at 
diagnostic methods employed by radiologists. 



5.1.1. Medical Background 

Bone age assessment is a procedure frequently performed in the pediatric 
radiology. Based on a radiological examination of skeletal development 
of a left hand wrist, the bone age is assessed and then compared with 
the chronological age. A discrepancy between these two values indicates 
abnormalities in skeletal development. This ex amin ation is universally used 
due to the advantage of simplicity, a minimum of radiation exposure, and 
the availability of multiple ossification centers for evaluation of maturity. It 
is an important procedure in the diagnosis and management of endocrine 
disorders serving subsequently as one index of therapeutic effect. 2 Being 
a useful procedure in the diagnostic evaluation of metabolic and growth 
abnormalities 3 it indicates as well the acceleration or decrease of maturation 
in a variety of syndromes, malformations, and bone dysplasias. 4 Bone age 
assessment procedure is referred to for patients with gonadal dysgenesis 5 
or while metacarpal sign occurs. 6 It is also consulted in planning an 
orthopedic procedure which will be used for the correction of angular 
deformities or abnormalities of length involving the vertebral column or 
long bones. 

Bone age assessment is based on an analysis of ossification centers in 
the carpal bones and epiphyses of tubular bones including distals, middles, 
proximals, radius, and ulna (Figure 1). Epiphyses ossify usually after the 
birth. With increasing of age the bony penetration advances from the initial 
focus (Figure 2a) in all directions (Figure 2b). Penetration continues until 
the edges of metaphyses are reached (Figure 2c). The strip between the shaft 
and the ossification center diminishes progressively (Figure 2d) in thickness 
until it disappears completely at the completion of growth (Figure 2e), when 
the epiphysis and metaphysis fuse into one adult bone. 
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Figure 1 . Hand wrist radiograph. 



Carpal bones are another part of the information source. In the early stage 
they appear as a dense pin point on a radiograph. While developing, they 
increase in size until finally they reach their optimal size and characteristic 
shape. In the developmental order of appearance 7 capitate (Figure 1) and 
hamate is followed by triquetrum and lunate. Then scaphoid, trapezoid, and 
trapezium occur. Finally, pisiform overlaps the already existing triquetrum 
and hamate. The girl’s development is noticeably more advanced and may 
differ from boy’s development as many as 3 years. 

Medical studies 8 indicate that, due to the carpal bones nature of maturity, 
their analysis does not provide accurate and significant information for 
patients older than 9-12 years of age. In this stage of development the 
phalangeal analysis yield more reliable results. Besides, more reliable results 
can be obtained while analyzing the most distal hand area. 4 

Numerous roentgen methods have been proposed for assessing the bone 
age according to the time of appearance, size, and differentiation of the 
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Figure 2. Ossification centers at different stages of development, (a) The epiphysis 
is distinct in appearance and disc-shaped with a smooth continuous border, (b) the 
epiphysis is half or more the width of the metaphysis, (c) the epiphysis is as wide as 
the metaphysis and follows closely its shape, (d) fusion of epiphysis and metaphysis 
has begun, (e) fusion of epiphysis and metaphysis is completed. 
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ossification centers. Different time schedules of skeletal maturation from 
fetal until adult life have been developed. The most commonly used method 
(76%) 9 is the atlas matching method by Greulich and Pyle. 10 Based on this 
method, a left hand wrist radiograph is compared to the series of radiographs 
grouped in the atlas according to age and sex. The pattern which superficially 
appears to resemble the clinical image is selected. Since each atlas pattern 
is assigned to a certain year of age, the selection assesses the bone age. 
However, there is still concern about the deviation in estimating the age that 
leads to a high rate of mis-classification. 11 The disadvantage of this method 
is a subjective nature of the analysis performed by various observers with 
different levels of tr ainin g. Studies have shown 12 inter-observer difference 
ranging from 0.37 to 0.6 years, whereas the intra-observer differences ranging 
from 0.25 to 0.47 years. The reason that atlas-matching methods feature such 
high discrepancies, results from a general comparison of the radiograph to the 
atlas pattern. By a more detailed comparison of individual bones, ambiguous 
results may be obtained. As an example, the development of carpal bones may 
lead to one result, whereas, the epiphyseal development may yield another. 
This causes the measures to vary from person to person and from observation 
to observation. 

Another technique relies on a trained observer applying the Tanner and 
Whitehouse (TW2) method. 13 This method uses a detailed analysis of each 
individual bone (epiphysis-metaphysis complex of each tubular bone and 
carpal bones). Each complex is assigned to one of eight classes reflecting the 
following developmental stage: 13 

stage B: single deposit of calcium, 
stage C: center is distinct in appearance, 

stage D: maximum diameter is- half or more the width of metaphysis 

stage E: border of the epiphysis is concave 

stage F: epiphysis is as wide as the metaphysis 

stage G: epiphysis caps the metaphysis 

stage H: fusion of epiphysis and metaphysis has begun 

stage I: epiphyseal fusion completed. 

This leads to the description of each bone in terms of scores. The sum 
of all scores assesses the bone age. This method yields the most reliable 
results. 11 ’ 14 Unfortunately, the complexity of this method prevents the rate 
of its application from exceeding 20%. 5,11 

The above presented remarks show that it is worthwhile to develop a 
computerized system to assist the radiologists in performing a more objective 
and accurate analysis yielding the bone age assessment. 
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Figure 3. Pattern recognition phases. 



5.1.2. General Computerized Approach 

Referring to human observations, in a classical pattern recognition approach 
three phases can be distinguished (Figure 3). Firstly, a preprocessing step 
standardizes the image in terms of the background and orientation. The 
background standardization removes parts of the image background caused 
by blocking of the collimator during the X-ray exposure, which do not deliver 
any pertinent information, adversely affecting observer performance and/or 
computerized image analysis. 15 The orientation procedure rotates images (if 
necessary) to a standard position viewed by radiologists and clinicians. The 
preprocessing part 16 refers to the very first action, done by radiologists or 
technicians, namely, to place the hard copy onto the illuminated panel (light 
box) at a standard position. Additionally, the computerized approach offers 
a reduction of background light. 

Secondly, the feature extraction step (Figure 4) develops measurement 
procedures which extract a set of parameters describing the object. This 
turns a natural object into a mathematical model which describes the object 
in terms of quantitative measures. In this approach three regions of interest 
(ROI) are selected: phalangeal region of interest (PROI), carpal bones region 
of interest (CROI), and epiphyseal regions of interest (EROIs). The analysis 
of certain regions depends on a rough estimation of the developmental stage 
of the patient performed on the basis of the length of tubular bones. This 
estimation (not being used directly in the final bone age assessment) indicates 
which ROIs are to be considered. The phalangeal length is extracted from 
the PROI whose analysis is performed for all patients. It delivers also the 
epiphyseal and metaphyseal diameters whose ratio is employed as a feature 
in the final phalangeal bone age assessment. The diameters themselves serve 
as edges of the EROI. For young patients, up to 9-years-old, the CROI 
analysis is initialized. For older patients carpal bones do not deliver pertinent 
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Figure 4. Flow diagram of feature extraction phase. 



information. The EROIs, although can be employed for all patients, are 
advised to be used for patients older than 9 years of age. For younger patients 
epiphyses are not fused and the extracted feature remains unchangeable. From 
each of the selected ROIs, certain features are extracted and passed to the 
next phase. 

Finally, in the classification phase (Figure 3), a transparent algorithm, 
developed on the basis of an opaque clinical method of radiological analysis, 
is employed. The algorithm is again split into three independent classifiers 
(one for each ROI). The initialization of each classifier depends on whether 
in the feature extraction phase a corresponding ROI has been analyzed. 
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This method, analyzing a hand wrist Computed Radiography (CR) image 
(Figure 1), turns it into a mathematical model (feature extraction phase) and 
compares with a mathematically developed standard to be able to assess 
the developmental stage. Since all regions are evaluated independently, the 
assessments reflect the maturity featured by each region individually. 

The computerized analysis still remains a compAter-assisted or computer- 
aided diagnosis. Due to the variety of human nature none of the computer 
system designer is able to foresee all possible cases and, thus, none of 
the computerized system is able to provide a 100% classification rate. 
T his is true not only for the computerized systems. Additional medical 
consultation is often required when an unusual case is diagnosed. Therefore, 
each automatically performed analysis can serve as an assistant in order to 
improve the examination terms discussed above. Yet, the clinician is still 
responsible for each final diagnosis. 



5.2. FEATURE SELECTION 

There is no general approach to the problem of feature selection. Features 
should provide an adequate description of the pattern and be sensitive to the 
structural changes on the basis of which an image is classified. Thus, features 
applied to assess the bone age should reflect the morphological changes which 
are able to distinguish one developmental stage from the other, for example, 
the increase in size of epiphyses, tubular bones, or carpal bones. They may 
also point out an object or a change of an object which appears at a certain 
stage of development. The more accurate a feature is able to describe the 
changes between certain stages, as a more reliable classification result can 
be obtained. If features do not reflect the structural changes, no classifier of 
whatever sophistication can yield a reasonable result. 

On the other hand, the features have to be extracted reliably (i.e. with 
a high rate of correct extraction) and cannot require too sophisticated and 
time consuming algorithms. It happens often that some features of a high 
discriminant power do not meet the above condition and cannot be extracted 
automatically. Looking at some features taken into consideration by the 
TW2 method, 13 these remarks become clear. As an example, let us compare 
features which distinguish stage F (Figure 2c) from stage G (Figure 2d) in the 
analysis of proximal phalanges. It is hard to think of a fully automated method 
which determines whether the epiphysis caps the metaphysis. The feature 
selection step is then a compromise between the features discriminant power 
and the ability and reliability of their extraction. Therefore, the computerized 
approach often uses features which reflect the changes, yet, which have not 
been implemented in the clinical diagnosis. 
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5.2.1. Phalangeal Analysis 

In the phalangeal analysis three types of features have been extracted. The first 
type includes the length of distals, middles, and proximals. While applying 
the standard phalangeal length table 17 each measure can be assigned to a year 
of age. However, they do not appear to be reliable bone age indicators ^nd 
are only used for a rough maturity assessment. 18 The second type measures 
the diameters of epiphyses and metaphyses. Since diameters themselves are 
sensitive to the size of the hand and its rotation along the forearm axis during 
the examination procedure, their ratios have been calculated. This reflects to 
a certain extent the classification done by Tanner and Whitehouse 13 which 
distinguishes stages B through F. This is also a finding compared by some 
radiologists while applying the Greulich and Pyle method. 10 In the clinical 
approach it is not a quantitative finding, yet, it is considered during the 
visual comparison of the radiographs with the atlas patterns. The third type 
of features, again used only visually in the diagnostic procedure, describes 
the stage of epiphyseal fusion (stages H and I). In our approach this feature 
is quantified and assigned to one of four classes (no fused, early fusion, 
advanced fusion, fusion completed). 



5.2.2. Carpal Bones Analysis 

Carpal bones (capitate, hamate, triquetrum, lunate, scaphoid, trapezoid, and 
trapezium in Figure 1) are another part of information source in different 
methods of bone age assessment. 8, 10,1 3,1 9 Their development differs from 
the long tubular bones. As explained in section 5.1.1, they do not appear all 
at the same time. Capitate and hamate appear in the first year of age, whereas, 
pisiform appears at the age of 8 for girls and at the age of 1 1 for boys. 20 

At the early stage of development, the carpal bones appear on the 
radiograph individually. Their separation and description in terms of features 
is relatively simple. However, at the age of 9- 1 0 they start overlapping. At this 
point, further analysis becomes very difficult. As mentioned in section 5.1.1, 
due to their nature, they do not deliver pertinent information for patients older 
than 9-12 years of age. At this stage of development the phalangeal features 
lead to more reliable results. Kirks says: 4 “ The more distal, the better". Thus, 
the carpal bones analysis is performed only for patients younger than 9 years 
of age. 



5.2.3. Evaluation of Extraction of Features 

The feature selection problem returns after the feature extraction algorithms 
have been developed. Two reasons may cause a selected feature to be 
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rejected from being used in the clinical application: firstly, the accuracy of 
its extraction, secondly, its discriminant power. 

In the accuracy of extraction two problems have to be taken into 
consideration. The first one includes mis-extraction of objects. This means, 
that instead of finding, for example, a particular carpal bone, a noisy object 
reflecting soft tissue has been extracted and considered by the algorithm as 
a carpal bone, or, due to an unusual position of the hand, a length that has 
been found is not the desired diameter but a measure of a different object 
(or bone). In this case the answer of the evaluation procedure is binary: 
correct/incorrect extraction, meaning, a correct or incorrect extraction of the 
measured object. The second problem in the accuracy evaluation refers to the 
accuracy of the measures themselves and is performed for all bones correctly 
identified. In this measurement evaluation, computer measures are compared 
with the manual measures. The conventional evaluation for measuring the 
significance of a difference of means is referred to as a f -test . 21 Assuming, 
that both distributions are thought to have the same variance, the following 
steps are required: 

- calculation of the mean difference m \ and m2 and the mean standard error 
sd, 

- computation of t — (mi + mi)!su 

- evaluation of the significance of the t value for Student’s distribution 
probability function 

The second aspect includes the evaluation of the discriminant power of each 
feature. It may eliminate some features as well as restrict the implementation 
of others. This evaluation can be performed by plotting clustered measures (or 
their normalized values) versus the classes which are taken into consideration 
in the classification stage. Then, certain statistical parameters (mean value, 
standard diviation, distances from the cluster centers, common cluster area, 
etc.) are calculated to evaluate the overlapping of clusters. Finally, certain 
rules (discussed in the next sections) are applied to evaluate the discriminant 
power. 

Another statistical approach , 22 uses the F statistic which is a ratio of the 
between class variation to the within class variation. A feature of a large 
between class variation and yet low within class variation giving a large F is 
a good discriminator. 



5.3. REGIONS OF INTEREST / 

As a region, we term an area within an image that contains a certain 
type of information of equal importance for further analysis. The meaning 
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“importance” can be evaluated as negative as well as positive. While talking 
about importance in a negative sense, we consider information that is of no 
pertinent context for further analysis. These areas are called regions of no 
interest (RONI). On the other hand, importance in a positive sense is referred 
to regions which include information to be considered in further processing. 
These areas are termed regions of interest (ROI). 

In our approach one RONI has to be excluded, namely, the unexposed 
image background, caused by blocking of the X-ray collimator and resulting 
in white borders surrounding the image field and adding a significant amount 
of unwanted transparent area. The removal of this area is done by turning to 
black all pixels from this area. Since the area is not of a rectangular shape, it 
cannot be removed by “cutting off’ the border, which leads to the restriction 
of the image size. The algorithm 23 first searches for the row edges of the 
radiation field. The row edges points are those with standard deviation in 
the desired direction exceeding an empirically determined threshold. Then, 
the edges are reconciled to find the comer points. From these comer points, 
straight lines boundaries are interpolated. Points outside the contour are set 
to black. 

After removing the undesirable information (Figure 5a), two ROIs are 
defined: phalangeal region of interest (PROI) and carpal bones region of 
interest (CROI). They are extracted from a thresholded image. Since at this 
point the entire image has to be scanned, a fast average thresholding technique 
is applied. It turns to 0 all pixel values below the average image gray-scale 
threshold and to 1 all the values above. This method can be successfully 
applied only after removing the unexposed background. Otherwise, the 
threshold is too high. The thresholding procedure yields a “glove” image 
(Figure 5b) which is then scanned in the horizontal and vertical direction. 

The upper edge of the PROI (Figure 6) is found by scanning a horizontal 
profile from the top of the image down to the tip of the third phalanx. The 
lower edge cuts the image at the soft tissue junction between the thumb 
and the index. Then, two vertical profiles are scanned from the left and 
right radiograph edges towards its central part. The first intersection with 
the “glove” defines the left and right PROI, respectively. The lower PROI 
edge serves also as upper CROI edge. The lower CROI edge is determined 
by scanning the forearm upward from the bottom of the image. The fine 
intersecting the forearm with the minimum width fixes the lower edge. Two 
vertical profiles are scanned similarly to the PROI profiles, yet, independently 
from them. Again, the first intersection between the profiles and the pixels 
within the hand contour marks the CROI boundaries. Though, the PROI 
includes distals, middles, proximals, and parts of metacarpals, whereas, 
the CROI includes all carpal bones, parts of metacarpals, parts of radius 
and ulna. 
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Figure 5. Hand wrist radiograph (left) and its “glove” version (right). 




Figure 6. Hand wrist radiograph with marked edges of the phalangeal region of 
interest (PROI), carpal bones region of interest (CROI), and epiphyseal regions of 
interest (EROI) of the HI phalanx. Corresponding EROIs are extracted for the II and 
IV phalanx. 



79 





234 



E. PIETKA and H.K. HUANG 



Within the PROI small epiphyseal regions of interest (EROIs) (Figure 6) 
are defined. They are limited by the epiphyseal and metaphyseal diameters. 
Though, they include the lower edge of the metaphyses and the upper edge 
of the epiphyses. Within this region the fusion analysis is performed. 

There is a structural difference between the PROI/CROI and the EROI. 
Although, all regions include bony structures displayed on the soft tissue 
background, the significance of the soft tissue is different in" both cases. 
In the PROI and CROI the bones to' be measured have to be separated 
(segmented out). Whereas, in the EROI the gap between the epiphysis and 
the metaphysis, being a soft tissue in the early stage of development is 
filled out by the bony structure while the fusion proceeds. Thus, it cannot 
be considered as a background in the sense, it is done in the PROI and CROI. 
Together with the edges of the epiphysis and the metaphysis the gap between 
the bones delivers equally important information. Therefore, the PROI and 
EROI, although, describing the developmental stage of the same region, are 
discussed separately. 



5.4. EXTRACTION OF PROI FEATURES 

The phalangeal region of interest (PROI) includes distals, middles, and 
proximals of all phalanges, as well as parts of metacarpals which are ignored 
in the approach. The first group of features reflects the developmental stage 
of phalanges, as far as their size is concerned. Each phalanx is described 
in terms of the phalangeal length and the ratio of the epiphyseal diameter 
divided by the metaphyseal diameter. All distance measures are extracted 
from a gradient PROI in which the bone edges are significantly enhanced. 



5.4.1. Gradient PROI 

Since the edges are in a continuous 2-dimensional domain f(x,y), they can 
be detected by a gradient G (/(*, y)) defined for the x and y coordinates as 



G (/(*,?)) = 



This means, that the gradient along the line normal to the edge slope is defined 
as derivatives along orthogonal axes. The vector G (/(*, y)), describing 
separately the row edge gradient df/dx and the column edge gradient 3// 3 y 
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can be presented in terms of a spatial gradient amplitude also called its 
magnitude 



G(/(x, y)) = 




which is the maximum rate of change of /(x, y ) per a unit distance in the G 
direction. For computation efficiency, the gradient may be approximated by 
the absolute value 

3 / 3 / 

G(/(x, y))= + j- 

dx dy 

Computation of the gradient is based on partial derivatives df/dx and df /dy 
computed at every pixel. For a 2 x 2 region 

Xl X 2 
X3 X4 

df/dx is computed by using a mask 

1 -1 
0 0 

and the computation of df/dy uses a mask 

1 0 
-1 0 

For a 3 x 3 region 

Xl X2 X3 
X 4 X 5 X6 
X 7 Xg X9 

df/dx is computed by using a mask 

-1 -2 -1 
0 0 0 
1 2 1 

and the computation of df/dy uses a mask 

-1 0 1 
-2 0 2 
-1 0 1 
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These two masks are often referred to as Sobel operators. Combination of 
both masks at any point results in obtaining the gradient at that point. A 
convolution of these masks with a region (image) yields a gradient at each 
point and is often referred to as a gradient region (image). The gradient 
computation can be based on a larger neighborhood. Then, it decreases the 
sensitivity to noise, yet, it decreases also the computational speed. Therefore, 
3x3 neighborhood is by far the most popular. 24 

The entire PROI is subjected to the gradient analysis. In the gradient PROI 
(Figure 7a) two anatomical regions are significantly enhanced. One covers 
the phalangeal edges, resulting in a clear separation between phalanges and 
the soft tissue surrounding the phalanges. The second dense region reflects 
the bone non-uniformity of epiphyses and metaphyses. As a result, these two 
regions concentrate the largest number of pixels of a high gradient. 



5.4.2. Extraction of Phalangeal Length Measures 

While analyzing the gradient PROI (Figure 7a) one can easily notice an 
increased concentration of pixels in the epiphyseal and metaphyseal area. 
It reflects the edges between epiphyses and metaphyses as well as the bony 
structure itself. An extraction of this region is necessary to separate the distals, 
middles, and proximals (in order to find their length) and to locate their 
extremities (in order to find the epiphyseal and metaphyseal diameters and 
extract the epiphyseal regions of interest for the analysis of fusion). For each 
phalanx, the separation is performed in three steps: 

- extraction of the phalangeal tip 

- rough extraction of the entire phalanx 

- extraction of epiphyses. 

These steps are performed interchangeably for all phalanges under 
consideration. 

Location of the phalangeal tip starts with the location of its candidate. The 
search is based on an analysis of profiles scanned from the top of the gradient 
PROI towards the bottom. Due to the length of fingers, the first tip to be found 
is the tip of the HI phalanx. Then, based on mathematical morphology, 25 
a dilation/intersection reconstruction sequence measures the object size. A 
contiguous set of non-zero pixels is dilated (described in Appendix A) and 
then logically intersected with the thresholded image to form an object. This 
procedure is repeated up to a certain number of iterations. It is interrupted in 
order to accelerate the performance. If it is smaller than empirically defined, 
it is considered as noise generated by processing the soft tissue with the 
Sobel gradient operator and is removed from the PROI. The search is then 
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continued. If the examined object is large enough, it is considered as the tip 
of the III phalanx. 

In order to roughly locate the entire HI phalanx, a predefined window 
covering the width of the phalanx (Figure 7b), is shifted from the tip over 
the entire phalanx (to the bottom of the PROI). Bone edges marked on the 
gradient PROI, serve as a basis to locate the phalangeal central axis. This 
permits each window to be centered, meaning, that the window central axis 
is intended to cover the bone central axis by shifting the window to the right 
or to the left depending on the rotation angle of the finger. 

The extraction of epiphyses (Figure 7c) is based on an analysis of the 
concentration of pixels in each window. The concentration is considered as a 
function versus the distance from the phalangeal tip. Since the concentration 
of the high gradient pixels increases significantly in the epiphyseal area, this 
function is searched for three local maxima. The distances between them 
cannot be smaller than a predefined value. Its assessment depends on the 
range of age considered in a certain application. It has to be smaller than 
the length of distals, middles, and proximals of patients belonging to the 
youngest group. This measure can be derived directly from the radiographs 
or assessed on the basis of the standard phalangeal length table. 17 If the 
distance measures do not meet the predefined conditions, certain correction 
steps 26 can be applied. 

Once the epiphyses are found the separation lines between the distals, 
middles, and proximals are extracted (Figure 7d). These lines are defined 
as short profiles (of a length of the predefined window) perpendicular to 
the phalangeal central axis and having the smallest intersection with the 
bone. Two cases are possible. For older patients whose epiphyses are already 
developed, one profile of minimum intersection with the bone is found. For 
younger patients a gap between epiphyses and metaphyses causes several 
profiles not intersecting the bone to be found. In this case, a profile tangential 
to the epiphysis marks the bottom of the epiphysis, whereas, a profile 
tangential to the metaphysis located below marks its top. The Euclidean 
distances between the already found intersection points serve as the length 
measures. 

Above the separation lines the left and right extremities of epiphyses and 
metaphyses are located. Based on the analysis of distances between the 
phalangeal central axis and the bone edge, three following points on the 
bone edge are detected one each side of the central axis: 

- maximum distance which reflects the metaphyseal extremity, 

- minimum distance which locates the narrowest area between the epiphysis 
and metaphysis (can be equal to 0 for very young patients), 

- maximum distance which reflects the epiphyseal extremity. 



83 




238 



E. PIETKA and H.K. HUANG 




Figure 7. Phalangeal region of interest (a) processed with the Sobel gradient 
operator, (b) The n, in, and IV phalanx overlapped with scanned windows (in each 
iteration the window is centered), (c) extraction of epiphyses, (d) separation lines for 
the II, m, IV phalanx, (e) extremities of epiphyses and metaphyses marked for the II, 
HI, and IV phalanx. 
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In the analysis, first, the bone edge is searched for the minim um Then, 
both maxima are found, one above and the other below the minimum. These 
points are located on the left and right bone edge (Figure 7e). The diameters 
of epiphyses and metaphyses are defined as Euclidean distances between the 
left and right extremities of epiphyses and metaphyses, respectively. 

Since distance measures are sensitive to the hand size, the length of fingers, 
and the rotation of the hand during the examination procedure, extracted 
values of phalangeal diameters for a certain year of bone age vary from one 
patient to another. While clustering the measures with respect to years of bone 
age, the neighbor clusters overlap significantly. To lower the sensitivity, ratios 
of epiphyseal diameters divided by metaphyseal diameters are calculated. 

These three steps (extraction of the phalangeal tip, rough extraction of 
the entire phalanx, extraction of epiphyses) performed on the III phalanx 
(as described above) are repeated for the II and TV phalanx. As a result, the 
following measures have been received: 

- length of phalanges (9 features), 

- ratios of epiphyseal diameter to metaphyseal diameter (9 features). 

Using the phalangeal lengths a rough assessment of the skeletal maturity 
is performed on the basis of the standard phalangeal length table. 17 The 
estimation associated with a single phalangeal length is obtained by a linear 
interpolation of the table values. This single estimators are then averaged 
to assess the global developmental stage. Unfortunately, it does not appear 
to be a sufficient indicator of skeletal maturity and can only be used as a 
rough estimate. Thus, it is used internally in the algorithm to limit the CROI 
analysis and initialized the epiphyseal fusion assessment. 

The ratios of the epiphyseal diameter to the metaphyseal diameter serve 
as a reliable indicator up to about 14 years of age. Due to the anatomical 
development of epiphyses, for older patients their discriminant power 
decreases. At this point the epiphyseal fusion becomes a crucial parameter. 
However, the PROI analysis is performed for all patients and the extremities 
of epiphyses and metaphyses indicate the EROIs. 



5.5. EXTRACTION OF EROI FEATURES 

Image analysis based on distance measures does not provide adequate 
information in the later stage of development, when the diameters of 
phalanges and epiphyses does not change. Epiphyseal fusion effects neither 
their length nor their width significantly enough to yield a reliable assessment. 
At this step the stage of fusion itself has to be assessed. 
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The analysis starts with an extraction of an epiphyseal region of interest 
(EROI) (as described in section 5.3). It contains the upper part of the epiphysis 
and the lower part of the metaphysis (Figure 6). The gap between both bones 
reflects either the soft tissue (before the fusion starts) or the bony structure 
(while the fusion proceeds). In this step a texture analysis is performed on the 
entire EROI, meaning, that all pixels included in the region are considered 
in the direct analysis. 

The EROI boundaries are defined by the diameters of epiphyses and 
metaphyses. The longest diameter determines the width of the region. Then, 
the EROI is rotated to align the metaphyseal diameter with the horizontal 
direction. This region is subjected to the wavelets decomposition algorithm. 



5.5.1. Wavelets Decomposition 

In this section, we introduce a 2-dimensional notation of the orthogonal 
wavelets representation. No proofs will be given. For more details we refer 
to 27 - 30 . 

A multiresolution approximation is a sequence of subspaces V_ m of L 2 (R 2 ) 



V m ^L+i formeZ 

n = t°) 

meZ 



and 



|^J is dense in L 2 (7? 2 ) 

mGZ 



The V_m+i subspace is derived from V_ m by scaling the approximated image 
(region or function) fix, y ) by the ratio of its resolution value 

fix, y) f(2x, 2 y) e V^ +1 for m € Z 

In this application we approximate an image at a 2 ; resolution (j e Z ). 
The approximation of an image fix, y) at a resolution 2 j is equal to its 
orthogonal projection on the vector space V_y- Let <p(x, y) € L 2 (R 2 ) be a 
scaling function such that if 02 j(^ ; y) = 2 2; 0(2 j jc, 2 J y), then form, n e Z 2 , 
2~i 4>2iix — 2~ in, y — 2 ~ j m) forms an orthonormal basis of V_ 2 j- 29 For 
a particular case of separable multiresolution approximation, the scaling 
function 4>ix,y ) can be written as <p(x, y) = <p(x)(piy) where (p(x) and 
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4> (y) are one-dimensional scaling functions of the decomposed subspace. Let 
be the one-dimensional wavelets associated with the scaling function 
(p (x). Then, we can build an orthogonal basis by scaling and translating three 
wavelets functions 



T^Cx, y) = 4>(x)f(y) 
3L 2 (x, y ) = tyix)$(y) 
V 3 (x,y) = 



The family of functions 



m,n e Z}(2~i foix — 2 - - / n)V r 2 -' 0 ? ~ 2~^m) 

^ 2 j(x - 2 - j n)<f> 2 j(y - 2~ j m) 
fvix - 2~ j n)f 2 j(y ~ 2 ~ j m)) 

is an orthonormal basis of complement of V_ 2j in V_ 2 i+ 1 27 The difference of 
information between the projection of /(x, y) at 2- /+1 and 2-> resolution 
is equal to the orthonormal projection of /(x, y) on this complement. 
Therefore, an image at a 2 J+1 resolution can be replaced by four images 
at a 2 j resolution. They are described in terms of a set of inner products 
which are equal to 2-dimensional convolution products evaluated at 2~ J n 



R% = (fix, y), *°(x, y)> = (fix, y), ^(x - 2 ~ j n, y - 2 ~ j m)) 
= ifix , y)*(h.ii-x)<h.ii-y))i2~ j n, 2 ~ J m) 

R\j = (fix, y), *\x, y)> = (fix, y), ^(x - 2 ~ j n, y - 2 ~ J m)) 
= ifix, y)*(h>i-x)f 2 ii-y))i2~ j n, 2 ~ J m) 

R\i = (fix, y), * 2 (x, y)) = (fix, y), tff,(x - 2 ~ j n, y - 2 ~ j m)) 
= ifix, y)*^ 2 J(-x)<p 2 j(-y))( 2 ~ J n, 2 ~ J m) 

= (fix, y), * 3 (x, y)> = (fix, y), 'I'JCx - 2 ~ J n, y - 2 ~ j m)) 
= ifix, y)*^ 2 ii-x)f 2 ii-y))i 2 ~ j n, 2 ~ J m) 
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There are different approaches to the scaling function 0 and construction 
function 0. One, used by Lemarie 30 starts from a multiresolution analysis 
framework. Two other approaches 27 ’ 28 start from a sequence h(n ). The 
summary of both approaches can be found in Appendix B. In this application 
the famil y of wavelets with compact support (derived in Appendix B) has 
been tested for N = 11, 13, and 15. 



5.5.2. Assessment of Epiphyseal Fusion 

The 2-dimensional wavelets decomposition algorithm, performed on a 2 7 x 2 7 
resolution region results in obtaining four images each at a 2 ;_1 x 2 7_1 
resolution. The , is the low frequency in both direction component, 

— the vertical high frequency component (horizontal edges), — the 
horizontal high frequency component (vertical edges), and i? 3 ^_, — the high 
frequency in both direction component. Each of these components is obtained 
by two one-dimensional convolutions of rows and columns separately with 0 
and 0 filters described in Appendix B. After the convolution only every other 
row and column is retained. This reduces the resolution from 2 j to 2 i ~ 1 . 

In a standard application of a wavelets decomposition algorithm, this 
process is a repeated predefined number of iterations. In following iterations 
only R^j-i components are convolved. However, in this study only the 
and Ry-i components are analyzed. The R^j-i component is tested in the 
preliminary step, whereas, the R\j~\ is a basis for the fusion assessment. It 
preserves the horizontal edges of epiphyses and metaphyses and, therefore, 
reflects the degree of fusion. 

Thus, subjecting the image region (Figure 8a) to the wavelets de- 
composition procedure, four components are received. They correspond 
to the combination of low and high pass filter applied to the original 
image interchangeably in the horizontal and vertical direction. The top left 
component is a result of the low pass filter convolved in both directions. This 
component preserves 99% of the energy. The top right component shows the 
results of the vertical high frequency and horizontal low frequency analysis. 
It enhances mostly the horizontal lines. The bottom left component is the 
horizontal high frequency and vertical low frequency component in which 
the vertical lines are enhanced. Finally, in the bottom right component the 
high pass filter has been applied in both directions. It shows high gradient in 
both direction points. 

While comparing the wavelets decomposition components of the EROI 
for patients at different developmental stages (Figure 8), one can notice that 
their density decreases while the epiphyseal fusion proceeds. Epiphyses (a) 
through (c) in Figure 8 are not fused. Fusion starts in (d) and (e), proceeds 
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Figure 8. Epiphyseal region of interest (EROI) arranged in the order of decreasing 
values of the horizontal low and vertical high frequency component (LHFC) energy. 
The left region presents the original EROI, the right region — the wavelets 
representation. The top left shows the low frequency in both directions component 
(LLFC), the top right — the horizontal low and vertical high frequency component 
(LHFC), the bottom left — the horizontal high and vertical low frequency component 
(HLFC), and the bottom right — the high frequency in both directions component 
(HHFC). The numbers describe the bone age assessment (in years) and the energy of 
the LHFC with respect to the energy of the original region. 
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in (f) through (h), and is completed in (i) and (j). A comparison of the 
percentage of energy calculated versus the energy of the original region 
of different components shows that the most sensitive component appears 
to be the horizontal low and vertical high frequency component (LHFC). It 
can be justified by looking at the structure of the developed epiphyses and 
metaphyses. While not being fused, their horizontal edges are included in the 
LHFC. The process of fusion effects mostly .this component decreasing the 
length of edges. For fused epiphyses the LHFC is close to 0. 

The boundaries between neighbor classes have been assessed by clustering 
the features individually into three classes: epiphysis not fused ((a) through (c) 
in Figure 8), fusion proceeds ((d) through (h)), and fusion completed ((i), (j)). 
T his results in obtaining the following values of the LHFC energy measure 
(EM). For non-fused epiphyses the EM ranges from 0.54% to 0.34%. While 
fusion begins, it drops to 0.22% and continuously decreases to 0.085%. For 
completely fused epiphyses the EM differs from 0.085% to 0.06%. Then, 
the interval of 0.23% to 0.085% has been divided into two classes which are 
considered as early and advanced stage of fusion. In the early stage of fusion 
the EM ranges from 0.28% to 0.18% ((d) through (f) in Figure 3) and in the 
advanced stage — from 0.18% to 0.085% ((g), (h)). 

Since the wavelets decomposition reflects the bony structure of the image, 
it appears to be sensitive to the radiographic exposure. An overexposed 
image enhances the structure of the tubular bones which appear also in the 
wavelet decomposition components (Figure 9). As a result, the energy of all 
components is significantly higher. Let us compare two LHFC derived from 
two images of a similar stage of fusion (Figure 9). The presence of additional 
line segments in the LHFC (Figure 9b in comparison with Figure 9a) reflects 
a non-uniformity of the bony structure that is of no interest in this application. 
Its amplitude overlaps with the amplitude along the edges of the epiphysis 
and metaphysis and cannot be eliminated. 

Therefore, before extracting the feature, employed to assess the stage of 
fusion, the LHFC is subjected to a histogram analysis. It serves as a quality 
control which prevents overexposed images from being analyzed. In the 
LHFC histograms of normally exposed images, a minimum value is followed 
by an increase of values reflecting the edges. While analyzing an overexposed 
image, the LHFC and its histogram is influenced rather by the structure of 
bones than by the edges. This results in a decrease of the histogram values. 
The amplitude reflecting the bony structure and epiphyseal and metaphyseal 
edges overlap and cannot be separated. The differences in the histogram 
shape defines the condition each image has to meet to be subjected to a further 
analysis. If the derivative of the histogram is negative in each point, the image 
is rejected. The second test applied to reject the overexposed image compares 
the high frequency in both direction components (HHFC in Figure 9). 
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VHFC=0.118% 

HHFC=0.030% 



(a) 




VHFC=0.564% 

HHFC=0.453% 



(b) 

Figure 9. Wavelets decomposition performed on (a) normal and (b) overexposed 
image. For the explanation of both regions see description of Figure 8. 



In a correctly exposed image it is much lower than in an overexposed image. 
Experimentally, limits are defined for each class of images. 

If none of these two tests reject the image its analysis is continued. The 
minimum of the histogram value serves as a threshold. Then, the energy 
of the LHFC computed over amplitudes exceeding the threshold value is 
calculated and divided by the energy of the original region (in %). This 
value is considered as the measure which indicates the stage of fusion. For 
non-fused bones, it is higher and is decreasing while the fusion proceeds 
(Figure 8). 



5.6. EXTRACTION OF CROI FEATURES 

5.6.1. Extraction of Bones and their Description in Terms of Features 

Carpal bones region of interest (CROI) includes (Figure 6) carpal bones, 
parts of the radius, ulna, and metacarpals. From this region two types of 
bones are extracted. Firstly, the carpal bones whose size is being described 
in terms of features, .secondly, the radius, which is subjected to the wavelets 
decomposition analysis leading to the assessment of the epiphyseal fusion 
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(as described in section 5.5 for the phalangeal epiphyses). An extraction of 
each bone is related to its separation from the background (the soft tissue). 
Due to a small discrepancy between bones and soft tissue (particularly in 
images of young patients), a 2-step thresholding method is applied. First, 
a rough separation is based on a histogram analysis calculated within the 
CROI. After smoothing the histogram, a local minimum following the first 
local maximum marks the gray scale value which serves as a rough threshold 
value. Then, a dynamic thresholding method 24 is applied. The thresholding 
value is calculated as an average gray scale pixel value within a window 
scanned over the entire region. Two window sizes are used. One, smaller (of 
9x9 pixels) for bones and larger (33 x 33 pixels) for soft tissue. A larger 
window, used for soft tissue pixels reduces the sensitivity to noise, whereas, 
the smaller window applied to pixels reflecting bones prevents the edges from 
being blurred. This procedure removes most of the soft tissue (Figure 10a). 
Yet, some objects (called later in this section noisy soft tissue objects) still 
remain. 

As a second step, a bone extraction procedure is applied. It is based 
on mathematical morphology. 25 Its task is to remove all non-carpal bones 
(radius, ulna, metacarpals, and noisy soft tissue objects). Since the radius, 
ulna, and metacarpals are attached to the CROI borders, they can be 
distinguished from carpal bones. This property is used for their extraction. 
The routine is based on a dilation/intersection reconstruction sequence. A 
non-zero pixel is dilated (described in Appendix A) and then logically 
intersected with the thresholded image to form an object. This procedure is 
repeated until the total area of the object does not increase in two subsequent 
iterations. This means, that the entire object is covered. This object is then 
removed by subtracting it from the CROI. This procedure is repeated for 
all non-zero pixels attached to the CROI borders. An opening operation 
(erosion/dilation reconstruction sequence) reduces some noisy objects. The 
current CROI (Figure 10b) contains carpal bones, epiphyses which are not 
attached to the radius, ulna, or metacarpals, and some noisy soft tissue objects. 
Their identification is based on features extracted from each of these objects. 

Therefore, the dilation/intersection reconstruction sequence, already per- 
formed for border pixels for now repeated in the entire CROI. After an 
object has been separated, a filling-holes routine is used to reconstruct the 
missing information. The routine consists of an iterating dilation/intersection 
reconstruction sequence performed on the inverse of the region which 
includes the object to be corrected. The routine stops after reconstructing 
the entire object. 

For each object the following parameters have been calculated: area, 
perimeter, compactness ratio, convexity coefficient, lengthening rate, gray- 
scale pixel value average, and gray scale pixel value discrepancy. 
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Figure 10. Carpal bone region of interest (CROI) (top) after the thresholding 
procedure, (centre) after removing bones attached to the CROI borders, (bottom) 
after removing all non-carpal bones objects and smoothing the carpal bones using 
mathematical morphology. 
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Area is the number of pixels included in the object. Perimeter is the 
number of pixels forming the contour. It has been obtained by subtracting the 
object from its dilated version. Compactness ratio is obtained by dividing the 
perimeter by the area and multiplying the result by a constant value. Center 
of gravity (CG) is found by eroding the object until the minim um number 
(>0) of pixels is left. Convexity coefficient is given by: 

ICC = P conv.hu \\/ P 

where P is the perimeter and P CO nv.hull is the perimeter of the smallest convex 
set which contains the entire object. 

Lengthening rate is obtained by dividing the smallest diameter by the 
largest one within the object. Grayscale pixel value average is obtained 
by referring to the gray scale image and computing the average of the 
corresponding pixel values. Discrepancy is obtained by subtracting the 
maximum and minimum pixel values of an object computed within the gray 
scale images. 

In order to separate the non-carpal bones, a polar coordinate plane is 
defined. The center of gravity of capitate, being the largest object of the largest 
average gray-scale pixel value serves as the origin. The CROI is then divided 
into five empirical sectors. Each may contain only one carpal bone or no 
carpal bone at all (for young patients). The multiangular bones section which 
contains two bones is the only exception. This division in based on the average 
anatomical location of the carpal bones. The center of gravity of each object is 
described in terms of the polar coordinates. The radius is a distance between 
the origin and the center of gravity itself, whereas, the angle is measured as a 
ray radiating from the origin to the center of gravity and the positive horizontal 
axis. Thus, determining the radius and angle of each object, all of them are 
assigned to one of the sectors. If more than one object belongs to a sector, 
described above features are tested against a predefined range of variability. 
The object, whose features meet this condition is considered as a carpal bone. 
All other objects are discarded. In the exceptional section, two carpal bones 
are accepted. 

Due to the anatomical order of appearance of carpal bones, the assignment 
can be doubly proved. If a carpal bone is missing, objects belonging to 
the corresponding sector are tested again within an enlarged range of each 
interval. As a result, from the CROI all non-caipal bones objects are excluded 
and carpal bones remain as solid objects (Figure 10c). 

As soon as the carpal bones are identified two additional parameters are 
computed to be used in the bone age assessment. They are: 

areafr 

ratio lb = 

area c 
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Figure 11. Original image with superimposed carpal bones extracted by the 
algorithm. 



where 



ratio2i, = 



area^ 

XX 1 area; 



where ratio 1^ and ratio2/ ; are parameters computed for the considered carpal 
bone, area,; is the capitate area, area^ is the area of the considered carpal bone 
and n is the number of carpal bones. 

Features extracted from selected normal and pathological hand radiographs 
have been published in 31 . Adding the extracted objects back to the original 
image highlights the detected carpal bones (Figure 11). 



5.6.2. Feature Selection 

After all carpal bones are defined, a 2-dimensional feature selection analysis 
is performed to compare their discriminant power in the bone age assessment. 
The following eight features are subjected to this analysis: area, perimeter, 
compactness ratio, convexity coefficient, lengthening rate, ratio 1, ratio2, and 
number of carpal bones. Features describing the location of carpal bones are 
not considered in further analysis. The number of features in each bone age 
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group depends on the number of carpal bones developed in a certain year of 
bone age. 

As reference images, a set of radiographs assessed clinically as normal, 
has been subjected to the feature extraction procedure. Then, each feature 
describing a carpal bone has been plotted versus the bone age (assessed 
by a radiologist). This gives a set of clusters. In order to exclude the most 
external measures, the set of observations for each year of bone age is fitted, 
to a Gaussian curve. The mean value m is assumed to be the center of the 
corresponding cluster, whereas, the variance S is adjusted by minimizing over 
8 the following expression: 

-y(xi;m,8)] 2 

i 

where 

y() is the Gaussian distribution 
m is the mean value 
8 is the variance 
yi are measures 
Xi are arguments. 

All values within the interval ±2<5 m i n remain in the cluster, others are 
excluded. In order to assess, whether a feature is suitable for further analysis, 
two discriminators are used: 



dist = mean, - mean l+ i 
distmax = distjmax,- + dist_max,-+i 

where dist_max ; - and dist_max, + i are maxima of feature distances form the 
corresponding cluster centers. 

A feature is selected, if dist > dist_max. 

Only four features (area, perimeter, ratio2, number of carpal bones) satisfy 
the above condition. Compactness ratio, convexity coefficient, lengthening 
rate, and ratio 1 have been discarded. 



5.7. CLASSIFICATION BASED ON A FUZZY CLASSIFIER 

Already described image processing techniques lead to a description of 
each image in terms of quantitative parameters. This turns the image into 
a mathematical model in which the developmental stage of phalanges 
and carpal bones is described individually. In the phalangeal region, first, 
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the length of tubular bones assesses roughly the developmental stage. This 
value is exclusively used by the feature extraction algorithm in order to 
initialize a certain option of further analysis. Then, the phalangeal diameters 
analysis is employed. This procedure delivers nine features (for distals, 
middles, and proximals of the n, HI, and IV phalanx). Next, the epiphyseal 
fusion is described in terms of a percentage of energy of a selected wavelets 
decomposition component in comparison with the energy of the original 
region. Finally, the carpal bones extraction is included, when the roughly 
assessed maturity does not indicate their overlap. In this stage the area, 
perimeter, and ratios of a single carpal bone area divided by the global area 
of all carpal bones are extracted for existing carpal bones. Due to the order 
of appearance the number of features change from one image to another. 
Once a mathematical model is derived, an algorithm which assigns the 
model to one of the predefined classes needs to be defined. The classification 
method depends on the discrepancy of the size and shape of objects (a 
single carpal bone, epiphysis, etc.) belonging to one class (intra-class 
variety) and to different classes (inter-class variety), discriminant power of 
features, and precision with which the clinical decision-making method can 
be described. 

The first two criteria may refer to two different levels of object definitions, 
firstly, to the object itself, secondly to its mathematical model. However, in 
order to estimate the inter- and intra-class variety by means of quantitative 
methods, objects have to be turned to mathematical models. Otherwise, we 
are only able to estimate the variety in terms of such intuitively understood 
abstracts like ‘large’, ‘small’, etc. Therefore, the assessment of the object 
variety is in fact the assessment of the discrepancy between features in the 
inter- and intra-class context. This is, how the discr iminan t power of features 
can be estimated. The variety of objects is assessed while clustering the 
features. In the most desirable case, we expect the features to be concentrated 
around the center of the cluster and the clusters not to overlap between each 
others. Unfortunately, it would be very unrealistic to expect such clusters 
while dealing with natural objects. Moreover, a certain feature may have a 
high discriminant power with an acceptable overlap between classes for a 
limited number of classes showing no discriminant power for others. This is 
particularly true while analyzing the development of a certain type of objects 
from earliest to the most advanced stage. We may be able to find features that 
describe a certain state existing from the earliest to the most advanced stages 
of development (e.g. epiphyseal fusion). Yet, there may also be features of a 
limited usage. Firstly, the difficulty of extraction may occur (as it can be seen 
in the carpal bones analysis), secondly, a feature may reflect developmental 
changes up to a certain stage (ratios of phalangeal diameters) or may occur 
at a certain stage of development (appearance of carpal bones). 
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Nature of features and the way they are extracted impose the first condition 
which the developed classifier has to meet. Due to the variety of human 
nature, the developmental process, as well as pathological changes, features 
may exceed the limits defined for each year of age or may not be available at 
all. This causes an overlap of clusters referring to neighbor classes. Besides, 
all features are extracted automatically with an accuracy of about 90%. This 
means, that on an average, one feature per image fails. This implies the first 
condition. The classifier cannot be sensitive to an abnormal or mis-extracted 
feature. 

Another criteria which influences the choice of the classifier refers to 
the clinical assessment which is performed by radiologists on the basis 
of the original radiograph. This defines, to a certain extent, the nature 
of the classification problem. The inter- and intra-observer discrepancies 
(described in section 5.1.1) indicate the bone age assessment problem as 
very imprecisely defined. Furthermore, the lack of an updated, medically 
approved set of hand wrist images (the medically used set has been collected 
in the 1950s from a highly selected population) contributes to the low rate of 
accuracy. 

The above described imprecision related to the feature extraction and 
classification is included in three types of inexactness, defined by Kandel. 32 
Generality features a concept applied to a variety of situation. This refers 
to the variety in human nature as well as the variety related to different 
developmental stage combined often with other pathological conditions. 
Ambiguity describes more than one distinguishable subconcept. Each year of 
patient age is considered as a separate class which causes the classification 
function to have more than one maximum. Vagueness occurs when precise 
boundaries cannot be defined and functions take values other than just 0 
and 1. As mentioned above, the clusters overlap leading to an ambiguous 
assignment of certain features to two classes. Since the above described 
recognition problem does not lend itself to a precise formulation, it requires 
imprecise techniques able to handle its inexactness. 

Before converting the clinical bone age assessment method into a 
computerized technique, describable in terms of an algorithm, the clinical 
method itself (considered as a source) needs to be understood. The most 
commonly used is the Greulich and Pyle method 10 which refers each 
image to an atlas pattern. This means that in the classification procedure 
each image is compared with patterns recognized medically as a standard. 
Since entire images are compared, the reference patterns have to be 
images of a corresponding structure. The process of comparison itself 
can be regarded as a personal (subjective) way of treating objects with 
individual judgments. These judgments are based on individual criteria for 
evaluation. In the bone age assessment this criteria can be defined as “image 
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resembles an atlas pattern”. However, while comparing two images, certain 
anatomical regions are compared separately. The Tanner and Whitehouse 
method 13 indicates certain anatomical changes very precisely and scores 
these changes, whereas, the Greulich and Pyle method is based on an overall 
comparison. A problem arises, when two regions of the diagnosed image 
(e.g. phalanges and* carpal bones), while being analyzed individually, are 
associated with two different years of age. Certain medical indicators can 
be found. Due to the “the more distal, the better” rule, the greatest accuracy 
is obtained 4 by comparing epiphyseal ossification of the distal phalanges 
with the standards. The relationship between epiphyses and metaphyses of 
distal phalanges may be evaluated by looking at the shape of epiphysis, 
modeling of epiphysis by adjacent metaphysis and bony structures, size of 
epiphysis^ and diameter of epiphysis compared with adjacent metaphysis. 10 
However, an individual criteria, very subjective and difficult to be defined is 
also possible. 

Following the medical evaluation scheme, in our classification approach, 
we refer to both steps, the comparison and the judgment. Thus, for the 
comparison, the mathematical model describing the image is referred to as 
a mathematical pattern which maps all features to comparable measures. In 
other words, features of different nature (area, perimeter, ratio, etc.) extracted 
from the image are mapped into a space of homogeneous measures. Then, for 
the judgment, a classification rule is defined to assess the bone age. Due to 
the inexactness described above, fuzzy techniques have been applied rather 
than “hard” classification algorithms. 33 



5.7.1. Membership Function 

The classification step is based on the fuzzy set theory introduced by Zadeh 34 
in 1965 which admits the possibility of partial membership of an object in a 
set called a fuzzy set. Let X = {x} denotes a space of objects (or features). 
Then, a fuzzy set A in X is a set of ordered pairs X = {(*, /j, A (x))}, x € X 
where /1 A (x) is a membership function. Intuitively, it defines the grade of 
membership of x in A. More precisely, it means that a membership function 
is a mapping from the set of features to a value from a predefined interval. 
We assume, that /x A (x) is a number in the interval [0, 1], The grade 1 denotes 
a full membership, whereas, 0 refers to a non-membership. 

As an example, let us consider a class of real numbers that concentrate 
around a real value R. We may define a membership function 

H(x) = 1/(1 + (x - R) 2 ). 
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Its assignment is subjective in nature and reflects the context in which the 
problem is viewed. The membership value is in the interval [0,1]. It increases 
while x comes close to R. Mathematical foundations of the fuzzy set theory 
can be found in 32 > 34 > 35 . 

The classification process leading to the bone age assessment is performed 
separately for the PROI, EROI, and CROI. Since the classification is per- 
formed individually, we received three assessments reflecting individually-the 
phalangeal development, epiphyseal fusion, and-carpal bones development. 
Thus, for all regions a membership function has been developed. It assigns 
the developmental stage of the region to a class. For the PROI and CROI each 
class is associated with a year of bone age, whereas, for the EROI four classes 
are predefined (no fusion, early fusion, advanced fusion, fusion completed). 
Since a fuzzy set theory is applied, the mapping does not deliver a binary 
value, meaning, that only one class is pointed out. Therefore, the mapping 
results in a set of membership values, each reflecting the grade-membership 
of the developmental stage in a certain class. 

Definition of membership functions is based on a membership operator 



X(a,b,x) = l/(l+a(x -b) 2 ). ( 1 ) 

This operator can be considered as a membership function for a single class 
and a single feature. Parameters a and b depends on the range of variability 
of the feature within each class and their inter-class discrepancy. Each input 
feature x is mapped into a membership value in the interval [0, 1]. The larger 
the membership value is the more the feature reflects the corresponding class. 
Using the membership operator defined in Equation (1), the membership 
function defined for all classes becomes a vector 



a(a, b, x) 



~ X(fl\,b\,x)~ 

_X(a n ,b n ,x). 



( 2 ) 



where n is the range of the year of bone age. 

The calculation of a,- and £>; (i = 1, . . . , n) is described later. 

Since the mathematical model describes each image by the means of 
multiple features including three ratio of phalangeal diameters, epiphyseal 
fusion for each phalanx in the PROI and area, perimeter, and ratios for each 
carpal bone in the CROI, Equation (2) is extended to reflect a multiple feature 
environment giving a matrix 

/3(a, b, x) = [a (a, b, xi] • • • a(a, b, x m )] (3) 



where m is the number of features. 
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Both PROI and CROI are multi-object regions. In the PROI the analysis is 
performed on the n, HI, and IV phalanx, whereas, in the CROI up to 7 carpal 
bones are taken into account. Thus, each region is described in terms of a 
feature vector y = [xi , . . . x*] and k is a number of objects. For a multi-object 
region, the membership function in Equation (3) is extended to 



At( a, b, y) = [j8(a, b, xi) • • • j8(a, b, x k )] (4) 

This format of the membership function is used unchangeably only for the 
PROI. Due to specific changes featured by the mathematical model of carpal 
bones and epiphyseal fusion, this function has to be adjusted. For the CROI, 
the adjustment is caused by the developmental order of appearance of carpal 
bones. Their development differs from the development of long tubular bones. 
In the early stages they appear as a dense pin point on a radiograph. While 
developing, they increase in size until finally they reach their optimal size 
and characteristic shape. In the developmental order of appearance (Figure 1) 
capitate and hamate is followed by triquetrum and lunate (at the age of 3—4). 
Then, scaphoid, trapezoid, and trapezium occur (at the age of 5-6). Thus, in 
the CROI analysis not all carpal bones exist in the early stage. As a result, 
not all features have to be present at each year of age. This means, that the 
number of membership function components in Equation (2) differs. Thus, 
a is defined as 



a (a, b, x) = 



0 

Xciflj- 1 ,*) 
X(aj,bj,x) 



'-X(an,b„,x)-1 



( 5 ) 



The membership function component Xc related to the year of the carpal bone 
appearance is defined as 



Xc(aj,x) = 1/(1 +ajx 2 ) (6) 

and is included in Equation (5). For the following components, the de- 
pendency of j from the object is related to the developmental order of 
appearance of carpal bones. Thus, for capitate and hamate j is equal to 1 
and 2, respectively, for triquetrum and lunate j is, respectively, 3 and 4, and 
for scaphoid, trapezoid, and trapezium - 5, 6, and 7, respectively. Vector (5) 
is substituted in Equations (3) and (4). 
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The third membership function classifies the epiphyseal region of interest 
(EROI). Due to the lack of unambiguous medical assessment based on 
epiphyseal fusion, the extracted features (percentage of energy) is mapped to 
four predefined classes: no fusion, early fusion, advanced fusion, and fusion 
completed. Thus, the membership function consists of four components. 
Following the increasing order of the features, the membership function 
component for the ‘fusion completed’ class is defined as 

Xitel. *) — 1/(1 + a \x 2 ) x > 0 (7) 



for the advanced and early fusion 

Xi(cti,bi,x) = 1/(1 +ai(x - bi ) 2 ) 



where i = 2, 3 for advanced and early fusion, respectively, 
whereas, for ‘no fusion’ 



X4(«4, b4,x) = 

Then, a is defined as 

a(a, b, 



1/(1 + <24 (x — £ 4 ) ) X < Z?4 
1 x > bi, 

Xitel.*) 

X2te2,^2,*) 

X3te3,^3,*) 

_X4te4,&4,*)_ 



( 8 ) 



(9) 



( 10 ) 



Substituting Equation (10) in Equations (3) and (4), the membership function 
for a multi-feature and multi-object environment is received. 

All parameters a,- and b, are found on the basis of a training set. Since the 
membership function serves as a reference for the diagnosed images, it has 
to be defined on the basis of hand radiographs diagnosed as normal. Thus, 
the standard set being referred to as a pattern has been carefully selected. 
Then, each image has been subjected to the feature extraction procedure. The 
extracted features have been clustered according to the year of bone age. Due 
to imprecise nature of problems, (as discussed earlier) the neighbor clusters 
overlap, particularly for older patients, when the growth of epiphyses and 
carpal bones slows down. Since the assessment of a,- and bi depends on the 
overlap of two neighbor clusters, these parameters are determined separately 
for each membership component. Therefore, corresponding features of each 
object (bone) within each region are selected over all images of a certain 
class. Obviously, clusters corresponding to the j'-th class overlap with cluster 
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corresponding to the j - 1 and j + 1 class. The overlap width wj is 
measured as 



Wj = dmax+Jmax — w c (11) 

» j 7+1 

where w c is the distance between the centers of clusters and d maxj , d max 7+ i 
are maximum distances between the features and the corresponding center 
of the cluster for clusters j and j + 1, respectively. 

If two neighbor clusters overlap, the center of the common area is found. 
If the overlap does not occur, a point equally distant from their boundaries is 
found. Two types of membership function components can be distinguished. 
One type reflects clusters, located in a neighborhood of two other clusters, 
the other one reflects external clusters with one neighbor cluster only 
(membership function components defined by Equations (6), (7), and (9)). 
As far as the first type is concerned, for each cluster two such boundary 
points are located (one on each side). They serve as crossover points dj and 
dj- 1 (membership function value is equal to 0.5). These yields to a set of 
equations 



X(a,b,dj) = 0.5 
X(a,b,dj- 1) = 0.5 



( 12 ) 



whose solution fixes the values of parameters a,- and bi for each membership 
function component. As an example, membership function components 
defined on the basis of distal ratios of patients between 8 and 13 years of 
age are shown in Figure 12a. 

The external cluster, having one neighbor cluster only, have also one 
boundary point. In membership function components defined in Equation (6) 
the carpal bone under consideration appears at a certain year of age. Thus, 
it is assumed, that bj is equal to 0 and the common area center serves as a 
crossover point dj in order to calculate aj 



Xc{aj, dj) = 0.5 (13) 

As an example, the membership function components defined for the area of 
the trapezoid are shown in Figure 12b. Since the trapezoid appears at the age 
of 6, four components are defined. Three of them are assigned to the age of 7 
to 9, respectively, whereas, the additional component is assigned to the. age 
of 6. Due to a large overlap between clusters in the group age 8 and 9, the 
last membership function component has been shifted (Figure 12b). 
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Figure 12. Membership function components defined on the basis of (a) distal 
diameters for 8- to 13-year-old patients, (b) areas of the trapezoids (in CROI). Each 
curve represents an age group. 
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Finally, for both external membership function components describing the 
epiphyseal fusion (Equations (7) and (9)) both bj parameters are assumed to 
be equal to the corresponding centers of clusters, whereas, the aj parameters 
are calculated based on the common area center as shown in Equations (9). 

The classification step refers to the comparison stage of the clinical bone 
age assessment. It results in a fuzzy mapping from a set of features X = {x} 
to the set 7 = {y} of years of bone age (for the PROI and CROI) or 
epiphyseal fusion assessment (for the EROI). This fuzzy mapping is a subset 
of characterized by a membership function that associates with each pair 
(x, y) its grade-membership function, being a value in [0, 1]. 



5.7.2. Classification Rule 

The second step in the medical evaluation of a hand radiograph is a judgment. 
Based on a comparison between the radiograph and atlas patterns, the 
radiologist has to decide which pattern appears to resemble the clinical 
radiograph. This judgment leads to the bone age assessment. 

In comparison with the clinical diagnosis, the computerized decision 
making process being based on a fuzzy binary relation Q, Q C X x Y 
between features X and the predefined classes Y points out the bone age 
reflecting the developmental stage of the region under consideration (PROI, 
CROI, or EROI). Due to the shape of the membership function components 
for each feature, the maximum of grade-membership values calculated 
over all classes indicates the corresponding bone age. Therefore, a global 
maximum operation calculated over all classes should also indicate the global 
bone age corresponding to the developmental stage of the ROI. Yet, the fuzzy 
relation maps more than just one feature to each year of bone age (or the state 
of fusion for the EROI). Therefore, we cannot apply a single maximum rule to 
assess the bone age. The maximum operation has to be preceded by another 
one which compresses all grade-membership values mapped to one class. 
Then, the composition of both operations may lead to the extraction of the 
desired bone age. Two such operators (giving two compositions) have been 
tested. The first one, called later a max-min rule performed on Equation 4 is 
defined as 



max 

y 



min /i(x, y) 

- X 



where x denotes the features and y denotes the years of bone age for the 
PROI and CROI or the state of epiphyseal fusion for the EROI. 

This means, that for each class a minimum grade-membership value is 
found. This gives one grade-membership value for each class. Then, the 
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maximum grade-membership value over all classes yields the desired bone 
age (or fusion state) assessment. 

The second rule, called later a max-sum rule is defined as 



max 

y 



^Kx,y) 



which means, that the minimum operator of the max-min rule is replaced 
by the summation over all grade-membership values belonging to one class. 
The maximum value calculated over all classes yields the bone age (or fusion 
state) assessment. 

As an example, grade-membership values of a selected image have been 
placed in a matrix (Table 1). They show the fuzzy relation between ratios 
of epiphyseal diameters divided by metaphyseal diameters and predefined 
years of bone age. Columns marked as dl through p3 stand for distal ratios 
(d 1 through d3), middle ratios (m 1 through m3), proximal ratios {p 1 through 
p3), respectively. These columns are followed by the result of the sum and 
min operation. The “age” column shows the years of bone age related to each 
membership value. Due to the shape of membership function components, 
the membership values should be close to 1 for the year of age which 
corresponds to the bone age and close to 0 for other classes. If the bone age 
is assessed as a non-integer value (i.e. 4 years and 7 months) the values close 
to 1 are distributed between both neighbor integer age values. If features 
are mis-extracted or are out of a predefined range caused by pathological 
conditions this regularity can be destroyed (i.e. membership values of dl, 
d3, and m 1 in Table 1). One can easily notice (Table 1) that the max-min rule 
is much more sensitive to these failures than the max-sum rule. Although 
in the presence of the out-of-range features the overall sum decreases, yet, 
it is still significantly higher than the neighbor values. Thus, in the final 
classification procedure, the max-sum rule is applied. 

Finally, the max-sum value is compared with its largest neighbor sum 
value. If the neighbor value is smaller than 8% (refers to one month on the 
age axis) this value is ignored. Otherwise, an interpolation (with the accuracy 
of one month) yields the bone age. This permits the bone age to be assessed 
with the accuracy of one month. 



5.8. RESULTS AND CONCLUSION 

The evaluation of final results obtained in the classification phase by assessing 
the phalangeal bone age (PBA), carpal bone age (CBA), and the stage of 
epiphyseal fusion (SEF) is preceded by the evaluation of the extraction 
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Table 1. An example of membership function values calculated for a hand image. 
Columns marked as d 1 through p3 stand, correspondingly, for features extracted from 
distals (d), middles (m), and proximals (p) of the II, HI, and IV phalanx. The third and 
second last columns show the results of the min* p(x, y) and y) operators, 

respectively. The last column marks the bone age assigned to each membership 
function. The patients chronological age is 10 years, the radiologically assessed 
bone age is 8 years and 5 months, whereas the phalangeal analysis results in 9 years 
(max-sum result 4.21). 



41 


42 


43 


ml 


m2 


m3 


pi 


^2 


p3 


sum 


min 


age 


0.08 


0.01 


0.02 


0.00 


0.04 


0.06 


0.03 


0.07 


0.22 


0.54 


0.00 


7 


0.23 


0.02 


0.03 


0.01 


0.15 


0.30 


0.09 


0.35 


0.94 


2.11 


0.01 


8 


0.94 


0.01 


0.02 


0.03 


1.00 


0.73 


0.61 


0.67 


0.20 


4.21 


0.01 


9 


0.26 


0.03 


0.06 


0.03 


0.15 


0.09 


0.41 


0.12 


0.06 


1.21 


0.03 


10 


0.09 


0.11 


0.34 


0.27 


0.11 


0.08 


0.12 


0.06 


0.04 


1.23 


0.04 


11 


0.01 


0.11 


0.96 


0.88 


0.04 


0.03 


0.02 


0.01 


0.01 


2.07 


0.01 


12 


0.05 


0.88 


0.34 


0.05 


0.00 


0.00 


0.01 


0.01 


0.01 


1.35 


0.00 


13 



of features. The feature extraction phase, first, locates the objects (epiphysis, 
metaphysis, capitate, hamate, triquetrum, etc.) subjected to the extraction of 
features, then extracts the features. Thus, in the evaluation analysis both steps 
are considered independently. Due to different types of extractions (objects or 
bones extraction and feature extraction), the evaluation methods differ as well. 
For the location of objects the accuracy measurement is binary. An object may 
be correctly or incorrectly located. It is assumed to be correctly defined if its 
identification permits a correct feature extraction. Each partial identification 
(e.g. part of an object has been erased by the segmentation procedure) leads 
to a mis-extraction of a feature. Therefore, the identification of such an object 
is considered as incorrect. In other words, an object is evaluated as correctly 
located if the entire measurement field of the object, meaning, all parts of the 
object used in the feature extraction procedure are found. Then, in the feature 
extraction evaluation, the accuracy of measures is evaluated. It is based on 
a comparison with another measurement technique. In our evaluation it is 
compared with measures extracted by fixing manually corresponding points 
on the soft copy display and measuring the distances between these points 
automatically. 

In the evaluation of the classification phase, first, the discr iminan t power 
of each feature with respect to the bone age are estimated. Then, the PBA 
and CBA are compared with the bone age evaluated by a radiologist and 
called later a radiological bone age (RBA) assessment. Both PBA and CBA 
are considered individually. Finally, the epiphyseal fusion is referred to the 
RBA assessment. 
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Based on this scheme, the accuracy evaluation is performed for: 

- detection of epiphyses and metaphyses 

- accuracy of phalangeal length 

- accuracy of epiphyseal and metaphyseal diameters 

- detection of carpal bones 

- accuracy of the carpal bone measures 

- discriminant power of features 

- evaluation of classifications for PROI, CROI, and EROI. 

The evaluation has been carried out on 1 20 clinical radiographs giving 1080 
epiphyses and metaphyses and 727 carpal bones to be located and analyzed. 

For the PROI analysis, each epiphysis with the corresponding metaphysis 
is considered as one object. Thus, a failure in detecting one of these two 
elements, resulting in an incorrect extraction of the corresponding ratio of 
diameters, is considered as an incorrect detection of the entire object. A 
correct detection of 973 objects (out of 1080) has been obtained giving a rate 
of 90%. 

While comparing the phalangeal length with a manual measurement, the 
mean difference of 0.02 mm with a mean standard error of 0.08 mm have 
been obtained. A paired t test has indicated a significance level of 0.05. The 
correlation coefficient has been 0.995. 

A comparison of the computerized and manual measurements of the ratios 
performed separately for distals, middles, and proximals shows, that the 
mean difference between these two methods is 0.02 mm, 0.006 mm, and 
0.003 mm, respectively, with a mean standard error of 0.038, 0.017, and 
0.022, respectively. A paired t test has indicated that a difference of 0.02 
given a standard deviation of 0.02 has been detected with a power of 0.85 
and significance level of 0.05. The correlation coefficient is about 0.95 for 
all cases. 

For the CROI, a correct detection of 689 carpal bones has been obtained, 
giving the rate of 94%. The size of the missing carpal bones has not exceeded 
3 mm in diameter. The overall comparison of the accuracy of measures has 
been based on a visual comparison of the detected carpal bones overlapped 
with the original image. A significant enlargement of the carpal bones area has 
been observed in 45 (6%) cases. It has been caused by soft tissue attached 
to the bones during thresholding the CROI. A mis-extraction of the area 
influences also the perimeter of the corresponding carpal bone. Unfortunately, 
causing an increase of the total carpal bone area, all ratios are influenced. 

The evaluation of the discriminant power is based on the analysis of clusters 
of features grouped for each year of bone age. The most external measures 
have been excluded by fitting the data to a Gaussian curve (as described in 
section 5.6.2). If the limited clusters still overlap, the feature is rejected. The 
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results are shown in Figure 13. Ratios of distals, middles, and proximals 
have been plotted in Figures 13a, 13b, 13c. For the CROI analysis, area 
(Figure 13d), perimeter (Figure 13e), and ratio of each individual carpal 
bone area divided by the total carpal bone area (Figure 13f) have met the 
evaluation condition (section 5.6.2). Other features have been rejected. 

Evaluation of the computerized phalangeal and carpal bone age assessment 
is based on an individual comparison of PBA and CPA withRBA (Table 2). 
Due to the inter-observer difference' of the Greulich and Pyle method applied 
by radiologists, the computerized bone age assessment is accepted as being 
correct if it does not differ from the RBA more than 6 months. Results indicate 
that the phalangeal parameters seem to be more reliable than the carpal bone 
features. This shows, that phalanges are more sensitive to the developmental 
changes than carpal bones, particularly in pathological cases. Moreover, a 
discrepancy of more than 6 months between the PBA and CBA has been 
noticed 36 reflecting a difference in the development between phalanges and 
carpal bones in abnormal conditions. 

The results of the stage of epiphyseal fusion (SEF) (Table 3) show, 
that a non-fused region is unambiguously separated from the regions in 
which the fusion has begun. The interval of the energy measure (EM) of 
this class (section 5.5.2., Figure 8), although needed to be extended to 
0.32%, is still separated from the EM of epiphyses in which the fusion 
has already begun (its upper limit has also been extended to 0.27%). This 
means, that a non-fused epiphysis is not classified into a stage of ‘fusion 
proceeds’ or ‘fusion completed’. A similar situation can be noticed while 
epiphyses are completely fused. However, due to different conditions during 
the X-ray exposure, in early stage of fusion, an epiphysis can be classified 
as not fused. An overexposed image exhibits a structural non-uniformity 



Table 2. Accuracy of computerized bone age assessment based on an individual 
comparison of CBA and PBA with RBA. 





differs from RBA 


differs from RBA 


differs from RBA 




less than 6 mth 


less than 1 yr 


more than 1 yr 


PBA 


75% 


19% 


6% 


CBA 


63% 


20% 


47% 



Table 3. Classification results for four stages of fusion. 



truth / results 


no fusion 


early fusion 


advanced fusion 


fusion completed 


no fusion 


100% 


0% 


0% 


0% 


early fusion 


8% 


92% 


0% 


0% 


advanced fusion 


0% 


14% 


86% 


0% 


fusion completed 


0% 


0% 


0% 


100% 
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(a) 




Figure 13. Features plotted versus the bone age assessed clinically (a) ratios of 
distal epiphyseal diameters to metaphyseal diameters, (b) ratios of middle epiphyseal 
diameters to metaphyseal diameters, (c) ratios of proximal epiphyseal diameters to 
metaphyseal diameters, (d) normalized area of carpal bones, (e) normalized perimeter 
of carpal bones, (f) normalized ratio of individual carpal bones area divided by the 
total carpal bone area. 



110 




BONE AGE ASSESSMENT 



265 




bone age 
(c) 




(d) 
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(e) 




(f) 
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of bones (particularly the metaphysis). While being subjected to the wavelets 
decomposition procedure, the resulting wavelet coefficients of both types 
(fusion and bone non uniformity) overlap and its separation is impossible. A 
significant overexposure effects the histogram and the energy of the HHFC, 
permitting the image to be rejected automatically. However, a moderate 
overexposure may not change the shape of the histogram, yet, increases the 
LHFC energy arid leads to a mis-classification. 

The general evaluation shows that 10% of images have been considered as 
overexposed and rejected. If not imposing the quality condition, all of them 
would be classified as non-fused (due to a high energy of the LHFC). The next 
6.7% are mis-classified. This gives a result of 83.3% of correct classification. 
While improving the quality of radiographs, the accuracy may reach 90%. 

In conclusion, the phalangeal length 17 applied to the standard phalangeal 
length table does not appear to be a sufficient indicator of the skeletal maturity 
and can only be used as a rough estimator in an early stage of the hand image 
analysis. The measure is influenced by the length of the fingers and the overall 
length of tubular bones. Therefore, the phalangeal length table is appropriate 
for American-born subjects of European derivation. It cannot be applied for 
African ancestry who are generally longer in measure bone size. 

The comparison of the atlas matching method with the PBA and CBA 
shows a better agreement and accuracy of the PBA. This indicates, that 
phalanges are more sensitive to the skeletal development reflecting the 
changes more accurately. This confirms certain medical approaches 4 ’ 8 ’ 10 ’ 13 
to the problem of the bone age assessment. 

The EROI analysis shows that some components of the wavelets repre- 
sentation can efficiently be used for the classification of epiphyseal fusion, 
supporting the bone age assessment, particularly, for older patients when the 
size of bones does not serve as a reliable indicator. At this step, the internal 
structure of the bones has to be analyzed. 

While being applied as a clinical routine, this software package requires 
standardized images to be used as an input. The standardization has to 
include the background, orientation, and exposure. The first two elements 
are corrected by the pre-processing routines, 16 whereas, the exposure is only 
tested during the EROI analysis. Due to the lack of correction, it effects the 
results of the overall analysis. 

The entire computer package, integrated as an application package in a 
clinical PACS (Picture Archiving and Communication System) permits the 
hand radiographs to be archived automatically. The growing database creates 
opportunities to collect and approve medically a set of radiographs which 
may confirm or improve the already existing set of radiographs, collected in 
1950s. This improved set of images may serve as a reference for the clinical 
bone age assessment. 
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5.9. APPENDIX A: MATHEMATICAL MORPHOLOGY - BASIC 
OPERATIONS 

Mathematical morphology is to provide a quantitative description of geomet- 
rical structures of an image. The general approach is to probe and transform 
an image object with different patterns of predefined shape called structuring 
elements. 

The basic operations are the erosion and dilation. The erosion can be 
viewed geometrically as a transformation set which shrinks the considered 
object. For a binary image (in this study only binary mathematical mor- 
phology has been used), an erosion of X by a structuring element B is the 
intersection of the translates of X by all elements of B and is defined as 

E(X, B)= 

beB 

where Xb is the translate of X by b and b is an element of B. 

Dilation is the union of the translates of X by all elements of B and can 
be expressed by: 

D(X,B)= |Jx fc 

beB 

Dilation is a dual operation with respect to the complement of erosion: 

D(X C ,B) = E(X, B) c 

When an object is eroded and then dilated, this operation is called an 
opening and is denoted by: 

0(X, B) = D{E{X , B), X). 

Similar, the closing 

C(X , B) = E(D(X , B), X) 
is found by first dilating and then eroding the image. 



5.10. APPENDIX B: WAVELET DECOMPOSITION -DERIVATIVES 
OF SCALING FUNCTIONS 

Mallat 27 reconstruction and decomposition algorithm uses the multireso- 
lution analysis while computing the filter operators. Following Lemarie’s 
definition of the scaling function 
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0(®) = 



"VE 2 «(<») 



and 



E^) = E 



k=—oo 



1 

(o) + 27T ) n 



the multiresolution approximation is based on cubic splines obtained by 
computing the 6-th derivative of 

V' 1 

E(a>) = 5 

2 4sin 2 (<w/2) 



Since 



4>{2co) = H(co)(p(co) 



H(a>) can be obtained as 



JJ( ., I E 2n M 

The Fourier transform of the corresponding orthonormal wavelet can be 
derived from 



^■(2o>) = G(co)4>(co) 



and is given by 






e 1(0,2 I E 2 nW 2 + 7r ) 

V E 2n (") E 2 «("/ 2 ) 



The coefficients of the impulse response /z(n) are computed from the transfer 
function. The impulse response g(n) is given by 

g( n ) — ( — 1) 1— (1 — n). 

Daubechies 28 has extracted the properties of the filter operator without 
reference to the multiresolution analysis. This permits the following set of 
conditions to be defined. 
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(1) Since 

^|/i(n)| < oo 

n 

^|g(n)| < oo 

n 

the operators 

( Ha)k = y ^h(n — 2 k)a n 

n 

( Ga) k = - 2 k)a n 

n 

are bounded. 

(2) The reconstruction and decomposition scheme will work if H*H + 
G*G = 1. 

(3) The orthogonality of subspaces requires that HG* = 0. 

(4) The final condition identifies G as a difference operator which cor- 
responds to a band pass filter and H as an averaging operator which 
corresponds to a low pass filter. Thus, we require 

5>(*) = C 

n 

^2g(n) = 0 
n 

It can be proven that C = \fl. 

If a sequence has finite length, then the corresponding basic wavelet has 
compact support. This means that there are values AT and N + for which 
h(n ) = 0 if n < AT or n > N + . 

Conditions (1)— (4) imply the following equation 28 

|waKv)| 2 + |mjv(v + tt)| 2 = 1 

with the following structure on m 

m N (y) = [ 0 . 5(1 + e iv )f Q N {e iv ) 

where Q is a polynomial such that 
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and R is an add polynomial. 

The above equation is solved by a polynomial P N of order N - 1 






where z = cos 2 v/2. 

In our application Q of minimal order has been chosen. This means that 
i? = 0 and \Q n (e iv )\ 2 = P N s\n 2kV - 

In order to find the following Riesz lemma is used. 

Let A be a positive trigonometric polynomial containing only cosines, 
Mv) = ZLo a n cos (nv). 

Then, there exists a trigonometric polynomial B(y) = Yl„=o b n e ' nv such 
that |5(v)| 2 = A(v). 

The proof can be found in 28 . 

This defines 

tv- 1 

Q N (e iv ) = q n e inv 



-\N N - 1 



m n — 



-Q + e' v ) 



N - 1 



X) qn{n)e mv =2 1/2 h N (n)e l 



J n = 0 



n=0 
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PERFORMANCE OF ASYNCHRONOUS TRANSFER MODE (ATM) LOCAL 
AREA AND WIDE AREA NETWORKS FOR MEDICAL IMAGING 
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Abstract — Asynchronous transfer mode (ATM) technology emerges as a leading candidate for medical image 
transmission in both local area network (LAN) and wide area network (WAN) applications. This paper describes the 
performance of an ATM LAN and WAN network at the University of California, San Francisco. The measurements 
were obtained using an intensive care unit (ICU) server connecting to four image workstations (WS) at four different 
locations of a hospital-integrated picture archiving and communication system (HI-PACS) in a daily regular clinical 
environment. Four types of performance were evaluated: magnetic disk-to-disk, disk-to-redundant array of 
inexpensive disks (RAID), RAID-to-memory, and memory-to-memory. Results demonstrate that the transmission 
rate between two workstations can reach 5-6 Mbytes/s from RAID-to-memory, and 8-10 Mbytes/s from memory- 
to-memory. When the server has to send images to all four workstations simultaneously, the transmission rate to 
each WS is about 4 Mbytes/s. Both situations are adequate for radiologic image communications for picture 
archiving and communication systems (PACS) and teleradiology applications. © 1997 Elsevier Science Ltd. 

Key Words: Image co mmuni cation. Asynchronous transfer mode (ATM), Local area network (LAN), Wide area network 
(WAN), Picture archiving and communication systems (PACS) 




INTRODUCTION 

A picture archiving and communication system 
(PACS) uses local area networks (LAN) to transmit 
images within a medical center, whereas teleradiology 
relies on wide area network (WAN) technology. In 
radiology applications, proprietary LAN can achieve 
up to 1 Gbit/s, whereas WAN is limited to 
1.5 Mbits/s (1, 2). Since LAN and WAN use different 
technologies (3, 4) this creates potential integration 
problems when both PACS and teleradiology ser- 
vices are required. The slower transmission rate in 
WAN is not because of inadequate technology but is 
due to the cost imposed by long distance carriers for 
high speed communication lines. The current concept 
in medical image communication is that no physical 
or logical boundaries should exist between LANs and 
WANs (5). Asynchronous transfer mode (ATM) 
technology satisfies this requirement. 

ATM transports data by dividing them into 
53 byte fixed-length cells. Each cell consists of 
5 bytes of header information and 48 bytes of data. 
The basic signaling rate of ATM is 51.84 Mbits/s 



•Author to whom correspondence should be addressed. Tel.: 
415-476-6044; Fax: 415-502-3217. 
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called OC-1 (Optical Carrier Level 1). Others are 
multiples of OC-1, e.g. OC-3 (155 Mbits/s) and OC- 
12 (622 Mbits/s) (6-8). In imaging applications, the 
user can use the standard TCP/IP (Transmission 
Control Protocol/Intemet Protocol) protocols for 
both LAN and WAN or application software 
containing TCP/IP like the DICOM (Digital Imaging 
and Communication in Medicine) (9). 

An early large scale ATM project for high speed 
data transmission was the BERKOM (Berlin Com- 
munication) testbed in 1986 (10). The project was 
terminated in 1991 because the ATM technology was 
not mature enough for such large scale implementa- 
tion. Another experimental project using ATM for 
high speed image communication was the B-ISDN 
(Integrated Services Digital Network) field trial at 
North Carolina. In this testbed, an OC-12 was used 
to connect several remote sites for radiation therapy 
planning application and an OC-3 for remote 
medical consultation (1 1). 

We set up an ATM OC-3 (155 Mbits/s) WAN 
and LAN testbed network, shown in Fig. 1, 
connecting the University of California at San 
Francisco (UCSF), Mount Zion Hospital (MZH), 
and the San Francisco VA Medical Center 
(SFVAMC) in the San Francisco Bay area for 
image transmission in 1994. Results demonstrated 
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Fig. 1. Wide Area Network (WAN) with ATM OC-3 (155 Mbits/s) Technology connecting UCSF, MZH, and 

SFVAMC in a star architecture. 



that both LAN and WAN can achieve up to 
60 Mbits/s between a SUN SPARC 10 and a 
SPARC 20 workstation (SUN Microsystems, Moun- 
tain View, CA) with FORE ATM switches and 
adapter boards (FORE, Warrendale, PA) with TCP/ 
IP protocols (5). 

Since then we have designed and implemented a 
large scale ATM OC-3 LAN and WAN connecting 
UCSF, MZH, and SFVAMC (12). Figure 2 shows 
the network architecture. This paper describes several 
experiments to measure the performance of this 
ATM LAN and WAN in a clinical environment. 

PERFORMANCE OF ATM LAN 

The purpose of this experiment was to inves- 
tigate the ATM performance in LAN connected 
with workstations using different types of compu- 
ters and operating systems. Referring to Fig. 2, 
while the hospital-integrated PACS was running in 
a daily routine clinical mode, we inserted additional 
TCP/IP communication processes in six worksta- 
tions using different SUN computers connected to 
the LAN at UCSF. Each workstation had its own 
specific functions in the PACS and was running 
different operating systems. Data were transmitted 
memory-to-memory continuously between two 
workstations. Transmission rates for both sending 
and receiving were measured separately at each 
workstation. The transmission rate in the experi- 
ments described in this paper is defined as the time 
required from when the data were sent to when the 
data were received. Table 1 shows the results. The 
workstations shown in each row represent the 



sending sites while workstations in the columns 
represent the receiving sites. Thus, for example, 
matrix (1,2) = 1.35 Mbytes/s means that the per- 
formance was measured by sending images from 
SPARC 5 running Solaris 2.3 operating system and 
received by SPARC 20 running Solaris 2.4. 

Based on this table, we observe that, for 
memory-to-memory transmission: 

1. The ATM LAN can achieve over 9.0 Mbytes/s 
between two SPARC 20s. 

2. For a given pair of workstations, the performance 
is different if the order of sending and receiving is 
reversed. This can be explained as follows. In 
TCP/IP protocols, the receiving site is responsible 
for managing the incoming flow of data. There- 
fore, a lower performance computer used in the 
receiving workstation will result in a poorer 
transmission rate compared with when it is used 
for sending. 

3. The performance depends on the type of work- 
station and the operating system used. 

The poor performance of SPARC 5 for receiving 
prompted us to replace all SPARC 5 workstations in 
our network. The inferior performance of the newer 
operating system Solaris 2.4 compared with the 
earlier version Solaris 2.3 was puzzling. The ATM 
manufacturer has since identified that the problem 
was in the ATM driver and it has been resolved. The 
measurements reported in the next section were 
obtained exclusively with the Solaris 2.4 operating 
system. 
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Siemens 




Fig 2 ATM LAN and WAN connections between UCSF, MZH, and SFVAMC. AUI/UTP: attachment unit interface/unshielded twisted pair. The 
components in the top row are all image acquisition devices, dotted lines are components not yet connected; the second row components are image 
acquisition computers; the ICU module is in the bottom row with a bracket.The dotted components are used for the measurements of ATM LAN and 

WAN. See also Fig. 3. (Modified from Fig. 3, Ref. (12).) 



Performance of ATM • H. K. Huang el al. 


















168 



Computerized Medical Imaging and Graphics 



May-June/1997, Volume 21, Number 3 



Table 1 . Performance of ATM LAN (unit: Mbytes/s) 



REC SEND 


SPARC 5 
(Solaris 2.3) 


SPARC 20 
(Solaris 2.4) 


SPARC 20 
(Solaris 2.3) 


SPARC 20 
(SunOS 4.1.3) 


SPARC 20 
(Solaris 2.4) 


SPARC 20 
(Solaris 2.4) 


Function of the 
WS in PACS* 


SPARC 5 (Solaris 2.3) 




1.35 


4.72 


4.45 






isg.ucsf 


SPARC 20 (Solaris 2.4) 


0.25 




9.07 


5.50 


4.63 


4.54 


ICU I 


SPARC 20 (Solaris 2.3) 


0.30 


1.57 




7.29 


1.72 


1.76 


isg.nr 


SPARC 20 (Sun OS 4.1.3) 


0.05 


2.60 


9.47 






4.76 


mr2.acq 


SPARC 20 (Solaris 2.4) 




4.17 


9.59 






5.39 


ICU 2 


SPARC 20 (Solaris 2.4) 




4.22 


9.65 


9.75 


5.45 




ICU 3 




isg.ucsf 


ICU 1 


isg.nr 


mr2.acq 


ICU 2 


ICU 3 





•isg.ucsf: ICU Quality Assurance workstation, 

isg.nr: Neuro radiology workstation. 

mr2.acq: MR2 acquisition computer. 

ICU 1,2,3: Three ICU workstations. 

SPARC: computer model. 

(Solaris 2.3): (software operating system). 



PERFORMANCE OF ATM LAN AND WAN IN 
ACTUAL CLINICAL ENVIRONMENT 

Experimental set-up 

In order to measure the performance of ATM 
LAN and WAN under actual clinical environments, 
we used the ICU (intensive care unit) PACS module 
in our PACS (13) (see Fig.’ 2, bracket in the lower 
row). Figure 2 illustrates the clinical environment 
and Fig. 3 shows the detailed connections (14). The 
ICU module at UCSF with three workstations and 



one server was already in clinical operation for over 
3 months before the experiment. SFVAMC did not 
have an ICU workstation, we installed a fourth 
workstation there for 4 weeks during the experiment 
in order to measure the ATM WAN performance. 
Table 2 shows the hardware and software used in 
these workstations. 

Types of measurement 

ATM performance depends on the workstations 
and their operating systems as well as the storage 




Fig. 3. The architecture of the ICU PACS module and the PACS controller used for the ATM LAN and WAN 
measurements. Five dual-monitor, 1600x1 280-resolution display workstations running ICU applications were 
used with two ASX-200 ATM switches. The ICU server has a 30 Gbyte RAID disk. The workstation at 
SFVAMC was connected to UCSF via Pacific Bell’s ATM switch in Oakland. The PACS controller acquired 
images from all acquisition devices and transmitted to the ICU server through the ASX-200 ATM switch and the 

Ethernet during the experiment. 
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Table 2. Hardware and software used in the ATM WAN and LAN measurements 



Hardware: 

ICU server 

SUN SPARC 20 

Dual-monitor with 1600x1280 resolution 

FORE S-Bus ATM host adapter card (OC-3: 155 Mbit s/s) 

Sun S-Bus Ethernet host adapter card (10 Mbits/s) 

30 Gbytes redundant array of inexpensive disks (RAID) 
ICU workstation 

Same as the ICU server except no RAID 
PACS controller 

Sun 690MP with four central processing units (CPUs) 
Small computer systems interface (SCSI) disks 
FORE ASX-200 ATM Switch (non-blocking, 24 ports, 2.5 Gbits/s) 
Newbridge ATM Switch (Pacific Bell, Oakland) 

Software: 

Solaris 2.4 operating system in the ICU server and workstations 

SUN OS 4.1.3 operating system in the SUN 690 MP 

C programming language 

TCP/IP communication protocols 

Unix socket communication 

Client/server applications 

Transmission speed between local disks and memory: 

SCSI— ‘Memory: 4.0 Mbytes/sec 

RAID (level 5)-*Memory: 9.0-20.0 Mbytes/sec 



devices used. Four common storage devices are local 
SCSI magnetic disk, remote SCSI magnetic disk, 
redundant array of inexpensive disks (RAID), and 
computer memory. For this reason, four types of 
measurements were obtained: local disk to remote 
disk; local disk to remote RAID; local RAID-to- 
remote memory; and local memory-to-remote 
memory. The measured total transmission time, 
TOTAL, is in general, a combination of five different 
times required during the transmission: 

TOTAL = Ti + T 2 + T 3 + T 4 + T 5 

where 

T\. sender disk (or RAID) to local memory, 
r 2 : sender local memory buffering, 

r 3 : ATM transmission time, 

T 4 : receiver local memory buffering, and 

T 5 : receiver local memory to disk (or RAID). 

Since the local disk input/output (I/O) speed 
(about 4 Mbytes/s) is slower than that of the ATM 
OC-3 155 Mbits/s (about 19.3 Mbytes/s), the trans- 
mission rate between two workstations, either both 
with magnetic disks, or one with a magnetic disk and 
the other with RAID, was dominated by T\ and/or 
T s . Therefore the measurement did not truly reflect 
the ATM performance. The I/O speeds of RAID-to- 
memory (about 20 Mbytes/s) and memory-to- 
memory (higher than 20 Mbytes/s) are in the same 
range or faster than the ATM signaling rate, there- 
fore the measurements between workstations from 



RAID-to-meisory, or memory-to-memory, would be 
a better measure of the ATM performance. 

Experiment 1: magnetic disk-to-magnetic disk 
with ATM and Ethernet. The first experiment was 
to measure the ATM performance from disk-to-disk 
and compare the result with that of the conventional 
Ethernet network (10 Mbits/s). Figure 4a shows the 
configuration. During regular clinical hours, we 
measured the disk-to-disk transmission rates of CR 
(computed radiography) images from the PACS 
controller’s disks via: (1) the Ethernet; and (2) the 
ATM to the ICU image server’s disks. For each 
individual image, the ICU server received two copies 
of the image, one over the Ethernet and the other 
over the ATM, from the PACS controller. 

The transmission rate for the PACS controller to 
send CR images to the ICU server were measured on 
a 24 h/day, 7 days/week basis. Approximately 
1 month of transaction data was analyzed. The 
performance measurements were 1247 and 
452 Kbytes/s for ATM and for Ethernet, respec- 
tively. This result indicates that even with the 
constraint of slow disk input/output speed, the 
ATM is almost three times faster than the Ethernet. 

Experiment 2: magnetic disk-to-RAID with ATM. 
The second experiment was performed using the 
configuration shown in Fig. 4b. The set-up is almost 
identical to experiment 1 except that the receiving site 
at the ICU server uses the RAID. The average 
transmission rate is 1 500 Kbytes/s. Results from 
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Fig. 4. Configuration used for ATM performance measurements, (a) magnetic disk-disk, ATM and Ethernet; (b) 

magnetic disk-RAID, ATM only. 



experiments 1 and 2 demonstrate that the ATM 
performance from disk-to-RAID is only about 20% 
better than from disk-to-disk. This is due to the 
limitation of the regular disk I/O speed which 
dictates the transmission rate despite higher I/O 
performance of the RAID. 

Experiment 3: RAlD-to-memory transmission. 
We used the ICU PACS module to measure 
RAID-to-memory and memory-to-memory trans- 
mission rates of the ATM. In these two experiments, 
five dual-monitor, 1 600 x 1280-resolution rCCT work- 
stations using Sun. SPARC 20 computers were 
interconnected via two ASX-200 ATM switches 
(Fig. 3). The ICU server distributed CR images 
over ATM to three ICU display workstations 
(Pediatrics, Medical Surgical, and Cardiac Care) 
within UCSF, and a workstation at SFVAMC. This 
image server uses a 30 Gbyte RAID for all ICU 
images. In this environment, the three ICU WS 
connections are in the LAN whereas the SFVAMC 
WS is in the WAN. 

RAID-to-memory transmission rates were mea- 
sured for transferring 80 CR images, each 8 Mbytes 
in size, from the image server’s RAID over ATM to 
individual SUN SPARC 20 system memory of the 
workstations. The communication processes allo- 
cated 64 Kbyte TCP window size and 128 Kbyte 
image buffer size for the data transfers. 

Table 3 shows the results from point-to-point 
and concurrent transmissions of data over the LAN 



Table 3. RAID-to-memory transmission rates (unit: 
Mbytes/s) 



Number of concurrent 
processes 




From the server to 


VA 


PED 


MS CARD 


! WAN WS 


5.2 






1 LAN WS 




5.8 




1 WAN & 1 LAN WSs 


3.31 


3.68 




1 WAN & 3 LAN WSs 


1.8 


1.8 


1.9 1.9 



and the WAN. The average point-to-point transmis- 
sion rate between the ICU server within UCSF was 

5.8 Mbytes/s and to SFVAMC was 5.2 Mbytes/s. 
These transmission rates correspond to transferring 
an 8 Mbyte CR image in less than 2 s. When the 
server had to send images to all workstations 
simultaneously, the rates deteriorated to about 

1.8 Mbytes/s. In all these measurements, the stan- 
dard deviations computed were less than 2%, 
insignificant to affect the accuracy of the average 
values. 

Experiment 4: memory-to-memory transmission. 
Memory-to-memory transmission rates were mea- 
sured for transferring 2000 data packets, each 
128 Kbytes in size, from the image server’s SUN 
SPARC 20 memory over ATM to the individual 
SUN SPARC 20 memory of the workstations. This 
amount of data is equivalent to 256 Mbytes or 32 CR 
images. Again, 64 Kbytes of TCP window size were 
used for the communication processes. 

Results from point-to-point and concurrent 
transmissions of data over the LAN and the WAN 
are shown in Table 4. The average point-to-point 
transmission rate between UCSF and SFVAMC was 
8.1 Mbytes/s and within UCSF was 9.8 Mbytes/s. 
These transmission rates correspond to sending a 40- 
slice (about 20 Mbytes) CT examination over LAN 
or WAN in less than 3 s. When the server had to 
send images to all four workstations, the perfor- 
mance deteriorated to about 4 Mbytes/s. 



Table 4. Memory-to-memory transmission rates (unit: 
Mbytes/s) 



Number of concurrent 
processes 




From the server to 


VA 


PED 


MS CARD 


1 WAN WS 


8.1 






1 LAN WS 




9.8 




1 WAN & 1 LAN WSs 


6.3 


7.2 




1 WAN & 3 LAN WSs 


3.7 


4.0 


4.1 4.3 
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DISCUSSION 

We have successfully implemented an ATM 
LAN and WAN network at UCSF and two affiliated 
hospitals, Mount Zion Hospital and San Francisco 
VA Medical Center in the San Francisco Bay area. 
The ATM, back-up by the Ethernet network, 
provides a comprehensive high-speed and reliable 
communication for radiologic images and related 
data over both LAN and WAN. Our experiments 
demonstrate that ATM can be used for both LAN 
and WAN with standard TCP/IP protocols without 
changing the physical network set up. As for the 
performance, with the RAID, the transmission rate 
can reach 5-6 Mbytes/s, and memory-to-memory, 8- 
10 Mbytes/s, about 25-50% of the ATM OC-3 
theoretical limits. 

From the results of our measurements, we 
believe that the transmission rates of ATM can 
satisfy most radiologic image communication 
requirements in PACS and teleradiology applica- 
tions (15-17). Hence, a digital communication net- 
work with ATM technology would be a suitable 
choice for large-scale_PACS involving both LAN 
and WAN. Ethernet, on the other hand, is inade- 
quate for image communication, particularly in a 
hospital-integrated PACS environment, because of 
the shared bandwidth characteristics in standard 
Ethernet. 

It is important to note that ATM technology 
treats both textual and image data identically if TCP/ 
IP protocols are used. The only difference is that the 
size of the image data file is normally much larger 
and therefore the data buffer used during commu- 
nication is also set to be larger. However, when the 
TCP/IP protocols are embedded in an application 
software, for example, in the DICOM communica- 
tion software (9), commonly used in radiology 
applications, the application software treats the 
textual data and image data differently. The applica- 
tion software has to validate the image data 
according to the DICOM format after the receiving. 
This process will increase the transmission overhead 
and reduce the transmission rate as compared to 
textual data which is a stream of bits of information. 
Therefore, for a hospital already using Ethernet with 
TCP/IP protocols for image transmission, it is not 
difficult to convert it to ATM for two reasons. First, 
the communication software for both technologies 
are almost identical, and second, no application level 
software needs to be changed. On the other hand, if 
the hospital is using DICOM for image communica- 
tion then the application level software in DICOM 
has to be modified. 



Factors affecting the throughputs of an ATM 
network include: (1) the computer and the operating 
system used in the workstation; (2) the number of 
simultaneous transmissions; (3) processing overhead 
of the network protocols (e.g. TCP/IP); (4) storage 
device used in the workstation; and (5) data buffering 
in the computer system and the ATM switch. Among 
these factors, improvement of network transmission 
protocols and storage speed are two major- .deter- 
mining parameters for optimization of ATM 
throughput performance. 

The use of ATM for LAN has no other costs 
except purchasing the ATM switches and adapter 
boards assuming that the fiber cables are already in 
place. On the other hand, using ATM for WAN 
requires a long distance connection that the user 
must pay to a carrier. The price depends on the 
distance between the sites as well as the availability of 
optical fibers from carriers. Currently the charge 
from long distance carriers for ATM is still 
prohibitively high, a major detrimental factor for 
using ATM in WAN applications. An alternative is 
to lease dark fibers from companies other than long 
distance carriers. In this case, WAN becomes _ygry 
much like LAN except that the cable distance is 
longer. The cost of ATM technology is decreasing 
very rapidly except for the use of long distance 
cables. It is important that ATM users form a forum 
to convince the long distance carriers that lowering 
the long distance cost so that more users can take 
advantage of ATM WAN technology for medical 
image transmission would be to their benefit. 

SUMMARY 

We introduced the ATM LAN and WAN 
technology for medical image applications and 
described some earlier experiments and testbed 
ATM networks by other investigators. The ATM 
LAN and WAN architecture at UCSF connecting to 
Mount Zion Hospital and the San Francisco VA 
Medical Center in the San Francisco Bay area are 
given. We described a preliminary experiment on 
measuring the performance of the ATM LAN. 
Results demonstrate that ATM LAN can achieve 
over 9.0 Mbytes/s memory-to-memory between two 
SUN SPARC 20 workstations. Four other experi- 
ments using a clinical ICU PACS module operating 
24 h/day, 7 days/week which consists of one server 
and four workstations all with SPARC 20 computers 
are detailed. The four experiments measure the ATM 
LAN and WAN performance of: disk-to-disk; disk- 
to-RAID; RAID-to-memory; and memory-to- 
memory. The result from the case of one pair of 
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connections confirms the performance obtained in 
the earlier ATM LAN experiment which is about 8- 
10 Mbytes/s. The transmission rates start to deterio- 
rate gradually when the server has to send images to 
different workstations simultaneously. In the case of 
memory-to-memory, each can still achieve 4.0 
Mbytes/s or about 2 s for a CR image. 
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APPENDIX 1 

LIST OF ACRONYMS AND DEFINITIONS 



ATM: 

ATM board: 

ATM switch (SW): 
BERKOM: 

PD. 

DICOM: 

Ethernet network: 

Gbit: 

HI-PACS: 

I/O (input/output) speed: 
ICU server: 

ICU: 

ISDN: 

Kbyte: 

LAN: 

Mbit: 



Asynchronous transfer mode 

A computer circuit board inserted to the workstation for ATM transmission 
A relay box to direct ATM transmission 
Berlin Communication 
Computed radiography 

Digital Imaging and Communication in Medicine, a standard image format and communication protocol for 
radiologic images 

A most popular computer network with 10 Mbits/s signaling rate 
gigabit 

Hospital-integrated picture archiving and communication system 

Time required to read or write data from a storage device to a computer memory 

An image server supplies images to ICU workstations 

Intensive care unit 

Integrated Services Digital Network, a most inexpensive way of transmitting digital data through WAN 
kilobyte 

Local area network 
megabit 
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Mbyte: 

MZH: 

OC-1: 

OC-3: 

PACS controller: 
RAID: 

SCSI magnetic disk: 
SFVAMC: 

Solaris 2.4: ^ 

SPARC 20: 

SUN OP: 

TCP/IP: 

UCSF: 

WAN: 

WS: 



megabyte 

Mount Zion Hospital 
Optical Carrier Level 1 

Optical Carrier Level 3 which provides a maximum signaling rate of 155 Mbits/s 
The control device of the PACS 
Redundant array of inexpensive disks 
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FOR IMAGE INTEGRITY BASED ON THE DICOM STANDARD 
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Abstract — The data quality and completeness of acquired images, which we refer to as integrity, is considered as the 
most important requirement in the image acquisition design of the Picture Archiving and Communication System 
(PACS). The Digital Imaging and Communications in Medicine (DICOM) standard significantly simplifies the task 
of acquiring radiological images from a DICOM compliant imaging system into the PACS. However, human 
interaction with the imaging system by changing the DICOM communication settings can result in missing images 
during the PACS image acquisition. A scheme based on the DICOM Query and Retrieve (Q/R) service class was 
developed to automatically identify and recover missing images. In addition, grouping sequential scanned images 
such as a CT and MR image series is another potential process that can miss images because of no indication of the 
end of series. Two methods are presented for determining the end of series and the pros and cons of each method are 
discussed in detail. Two experiments in a real clinical environment were conducted; one with and one without the Q / 
R implementation. The statistical results indicate two highlights from this work. First, the Q/R scheme faithfully 
recovered all missing images caused by human interaction with the DICOM compliant imaging system. Second, 
there was no single image slice missed when grouping slices into a series using the presented grouping algorithm in 
the two experimental periods. © 1997 Elsevier Science Ltd. 

Key Words: Picture archiving and communication system, Digital imaging and communications in medicine, Image 
acquisition. Image integrity. Fault tolerance, Magnetic resonance imaging 




INTRODUCTION 

DICOM compliance among manufacturers is a 
major factor contributing to the interoperability of 
different medical imaging systems in a hospital 
environment (1, 2). The DICOM standard developed 
by the American College of Radiology and the 
National Electrical Manufacturers Association 
(ACR-NEMA) defines the basis of the interface 
mechanism allowing different manufacturers’ digital 
imaging systems to communicate images (3-5). With 
the standardized interface mechanism, the task of 
acquiring images from the DICOM complaint 
imaging systems to PACS and/or multimedia inte- 
gration systems is much easier than from those with a 
“closed” architectural design (6, 7). Figure 1 shows 
the schematic diagram of the PACS image acquisi- 
tion. The design of the PACS image acquisition task 
must consider the automatic recovery from failures. 
Failures may be a hardware malfunction, a software 
error, or a manual interruption during the PACS 
acquisition process. Methods for automatically 
detecting and recovering hardware and software 
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malfunctions have been discussed in our previous 
work (6). In the case of acquiring images from the 
DICOM compliant imaging system, interruptions are 
generally caused by human interaction with the 
imaging system. Regardless of failure type, the 
consequence is always the same; some images will 
be lost. 

From our experience, one characteristic of the 
DICOM compliant imaging system is that it allows 
image communication settings to be manually 
changed whenever required. All DICOM service 
classes require registration (or configuration) for 
communications. In an imaging system, this config- 
uration is usually implemented as a communication 
setting by maintaining a list of communicating peer 
and method (either image sending or receiving). 
Typically, one communication peer and method at a 
time can be selected for image communication. With 
this implementation, the operator of the imaging 
system can easily change the communication setting. 
This flexibility, in fact, is one of the main factors 
causing PACS image acquisition failures. For 
example, if the manual change occurs while images 
are being transferred to the PACS acquisition system, 
the image transferring process will be interrupted. 
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Fig. 1. A schematic diagram shows that the PACS image acquisition involves an interface mechanism to connect 

the imaging system. 



Furthermore, if the operator waits too long or does 
not resume the PACS communication setting, the 
images will have to be manually re-transmitted or 
they will be lost. Suffice it to say, neither manual 
transmission nor the loss of images is acceptable 
under the automated PACS image acquisition design. 

For cross-sectional images such as CT and MR, 
acquiring a complete series of image slices covering 
the body region of interest is important. In order to 
group multiple images into a series entity, the end of 
series, which indicates the completeness of sequential 
image scans, must be known. Unfortunately, there is 
no indication for the end of series in most imaging 
systems today including those that are compliant 
with the DICOM standard. This has been a problem 
for the PACS acquisition task. To circumvent this 
problem, the algorithmic determination of the end of 
series has to be integrated into the PACS acquisition 
process. Note that this task is far from trivial. An ill 
chosen algorithm can result in missing images and/or 
slow acquisition performance. 

We first present the concepts of an automatic 
PACS image acquisition process based on the 
DICOM standard. In section 2, the configuration 
of the image client and image server along with the 
acquisition database dedicated to the PACS image 
acquisition are described. Two possible methods for 
the algorithmic detection of the end of series, which 



are combined and implemented in our image 
acquisition process, are presented. In section 3, the 
performance and results of the preliminary experi- 
ment using the DICOM compliant PACS system are 
reported. We also discuss the situation when the 
image acquisition operation is interrupted during 
transmission and its consequences. In section 4, a 
comprehensive scheme based on the DICOM Q/R 
service class for automatically recovering lost images 
is described. Results from the second experiment 
using this DICOM compliant PACS acquisition 
system with the integrated Q/R recovery scheme are 
also given. We summarize this research by reviewing 
the performance of the implemented acquisition 
system and present suggestions to PACS developers 
and manufacturers of the imaging systems. 

CONCEPT OF DICOM-BASED PACS IMAGE 
ACQUISITION 

Hardware configuration and software packages 

Figures 2 and 3 show the hardware configuration 
and software program modules used in this study, 
respectively. The configuration includes one General 
Electric Medical Systems (GEMS) (Milwaukee, 
Wisconsin) Signa 5.4 MR scanner and one Sun 
SPARC LX (Sun Microsystems, Mountain View, 
California) for the PACS acquisition computer. 



Digital Communication Network (e.g. Ethernet) 

I 




g upper layer protocol for 
o n ! 

u 5 i 

Q ^ i 



DICOM 
C-Store Client 

Imaging System 



DICOM 
C-Store Server 
PACS 
V Image Acquisition 



Data 

Storage 



Fig. 2. The diagram shows the computer systems used in this study and the interface mechanism includes 
Ethernet and the DICOM C-Store server and client programs developed according to the DICOM upper layer 

protocol for TCP/IP. 



129 





Automated PACS image acquisition • S.L. Lou et al. 



211 




Fig. 3. The diagram shows the major software modules 
running in the imaging system and the PACS acquisition 
computer which form the PACS acquisition task chain. 



These two computer systems are networked via 
Ethernet. The program modules consist of the 
Central Test Node (CTN) software package devel- 
oped by the Mallinckrodt Institute of Radiology (St 
Louis, Missouri), and our own PACS image acquisi- 
tion software package. The image transfer from the 
MR scanner to the acquisition computer is based on 
a client-server communication paradigm of the 
ICOM Composite-Store (C-Store) service. The C- 
Store client process runs on the MR scanner to send 
images to the PACS acquisition computer. The task 
of receiving images is performed by the C-Store 
server process (one of the CTN example programs) 
running on the PACS acquisition computer. The 
formatting process groups the received image slices 
into a series file and standardizes every pixel of the 
images according to a predetermined byte ordering 
(e.g. big-endian or little-endian). The sending process 
transfers the formatted image series file to one 
component of the PACS which can be either an 
image -display workstation or an image archiving 
system. These processes form the task chain of the 
PACS image acquisition (6). 



PACS image acquisition database 

The PACS acquisition database plays an impor- 
tant role in the entire acquisition task chain at the 
acquisition computer. It serves three major functions. 
First, it supports the transaction of each individual 
image slice received by the acquisition computer. 
Second, it monitors the status of the received studies 
and their associated series processed by each 
program module of the acquisition .task chain. 
Third, it provides the basis for the automatic image 
recovery scheme to detect unsuccessfully transferred 
images which will be described in section 4. In 
summary, the database provides a foundation for 



easily maintaining the integrity of the acquired 
images. 

In general, an MR examination study includes 
multiple series and one series may contain more than 
one image slice. Based on this hierarchy, three 
database tables are used in our PACS acquisition 
design: study, series, and image. Each table contains 
a group of records and each record contains a set of 
useful data elements. For the study and series 
database tables, a unique data element as the primary 
key is preferred for accessing a particular record in a 
timely fashion. We use the study name and the series 
name as the primary keys for the two database 
tables, respectively. In addition to the primary keys, 
the following major data elements are recorded: (1) 
patient name and hospital identification number; (2) 
dates and times when the study and series were 
created in the imaging system and when they were 
acquired in the acquisition computer; (3) the number 
of acquired slices per series; (4) the acquisition status; 
and (5) the DICOM unique identification (UID) 
value for the study and series. The detailed descrip- 
tion of the image transaction database table will be 
presented in section 2.3.3. 



Determination of the end of image series 

The images of the series can only be grouped 
together by the formatting process when the end of 
series is determined by the image receiving process. 
To algorithmically determine the end of series in a 
manner both accurate and timely efficient is not 
trivial. We present two methods used for determining 
the end of series and discuss the advantages and 
drawbacks of each as follows. 

Presence of the next consecutive series. The first 
method for detecting the end of series is based on the 
presence of the next consecutive series (PNCS). This 
method assumes that the total number of image slices 
for the current series will be transferred to the 
acquisition computer before the next series begins. In 
this case, the presence of the first image slice of the 
next consecutive series indicates the termination of 
the previous series. The success of the PNCS method 
depends on the following two criteria: (1) the imaging 
system transfers the image slices to the acquisition 
computer in order; and (2) no image slices are lost 
during the transfer. Note that the first criterion 
depends on the specific implementation of the image 
transfer by the imaging system. If the imaging system 
transmits the image slices in sequence (e.g. GEMS 
MR Signa 5.4), the PNCS method can faithfully 
group the images of a series intact. On the other 
hand, if the imaging system transfers the image slices 
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in a random fashion (e.g. SIEMENS MR Vision), the 
PNCS method may conclude the end of series 
incorrectly. Even though one could verify the 
second criterion by checking the image slice order, 
whether or not the last image slice has been 
transferred remains unknown to the acquisition 
computer. Another drawback of the PNCS method 
relates to the determination of the last series of a 
particular study which is based on the presence of the 
first image slice of the next study. The time delay for 
this determination could be lengthy because the next 
study may not begin for hours. 

Number of consistent acquired images for a time- 
interval. The second method for determining the end 
of series is based on a time-interval criterion. The 
hypothesis of this method assumes that an image 
series should be completed within a certain period of 
time. With this method, the end of series is 
determined when the acquisition time of the first 
image slice plus a designated time interval has 
elapsed. This method is obviously straight forward 
and simple, but with a static time interval criterion 
chosen a priori, it is not dynamic enough to be 
practical in a clinical environment. Thus, an alter- 
native recourse is preferred in terms of the number of 
consistent acquired images for a time-interval 
(NCAITI). 

The NCAITI method requires one to store the 
number of acquired images for a given series at two 
different times, say time t\ and time t2 = t\ + At, for 
some predetermined constant At. By comparing the 
number of slices acquired at time rl vs the number of 
slices acquired at time t2, a premise is constructed for 
determining whether or not the series is completed. If 
the number of slices is constant, we conclude the 
series is completed, otherwise, we conclude the series 
is not yet completed. This process (time-interval with 
image slice verification) is iterated until a consistent 
number of slices has been reached. General issues of 
this method in regards to the length of At and 
whether At should be constant or variable are core 
parts of the NCAITI method. 

One argument in favor of the NCAITI method 
in terms of the variable At is that for a given series, 
the number of slices transferred from the imaging 
system to the PACS acquisition computer tend to 
decrease while the iterations of the NCAITI method 
increase. Therefore it seems reasonable to believe 
that At may be reduced proportional to the number 
of increasing iterations of the algorithm. A counter 
argument of this observation is that the number of 
slices transferred to the PACS acquisition computer 
may vary with time depending on the imaging system 
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design. For example, an imaging system may be 
designed to transfer image slices according to the 
imaging system workload and priority schedule. If 
the image transfer process has a low priority, then the 
number of slices transferred during a period when the 
system workload is heavy will be less compared with 
when it is light. In this case, usrng a constant At is 
more appropriate. Despite whether At should be 
constant or variable, the first assigned At is crucial 
because a short At may result' in missing images 
whereas a long At may result in a lengthy and 
inefficient image acquisition. 

Usually, the first At chosen for the NCAITI 
method is empirical dependent upon the imaging 
protocols used by the imaging system. For example, 
if the imaging protocols frequently used to scan 
patients generate a lot of images per series, then the 
At should be long, otherwise, a shorter At is 
preferred. However, it is possible for the NCAITI 
method to conclude a series is completed before it 
really is. This is because in some rare cases the 
operator or clinician is required to interrupt the 
scanning process in the middle of a series to conduct 
a patient position alignment or to inject contrast 
agent. If the time it takes to conduct such procedures 
is longer than the At, then the images scanned after 
the procedure will not be grouped into the current 
series. For a careless PACS design, this could result 
in a severe problem — missing images. 

For our PACS image acquisition design, the 
NCAITI method and the PNCS method were 
combined to determine the end of series for the 
GEMS MR scanner. We set the At equal to a 
constant value of 10 min. In addition, we imple- 
mented this task in the following stages: (1) identify 
and count the acquired slices for a particular series; 
(2) record the time stamp whenever the number of 
acquired slices has changed; and (3) update the 
acquisition status of the series. In order to do all of 
these, we need to track the acquired image slices 
using a transaction table designed in association with 
the series database table. 

Image transaction table. Whenever a DICOM 
image slice is acquired from the imaging system, a 
record with a designed PACS slice name is created in 
the transaction table. The PACS slice name is 
composed of the modality type, the imaging system 
identification, the study number, the series number, 
and the slice number. The task of identifying and 
counting slices can be accomplished by using the 
image slice records in the transaction table. Three 
major events are recorded in the transaction table: (1) 
the number of acquired slices; (2) the comparison 
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time when the number of acquired slices are 
compared with the previous iteration; and (3) the 
acquisition status declaring the series as stand-by, 
ongoing, completed, or image missing. Here, the 
information regarding the number of acquired slices 
and acquisition status is useful for the maintenance 
of the image receiving process. If the series is still 
ongoing, the comparison time is updated for the 
verification of the number of slices during the next 
iteration of the NCAITI algorithm. After the 
NCAITI algorithm detects the end of series, it can 
be further verified by the PNCS method. If the next 
series exists in the series database table or the next 
study exists in the study table, the image series is 
determined complete. Otherwise, one more NCAITI 
iteration is given and the series remains in stand-by 
status. In this case, the end of series will be concluded 
in the next NCAITI iteration regardless of the 
existence of the next series or study. In this way, 
the completeness of an image series is verified by 
both methods and the potential lengthy time delay 
problem of the PNCS method is minimized. 

PRELIMINARY EXPERIMENT 

The PACS acquisition system was tested in a 
clinical environment for the acquisition of GEMS 
MR images via the Ethernet and DICOM C-Store 
service. In the testing environment, there were six 
other CT and MR imaging systems along with one 
volumetric image display workstation connected to 
the same network (8). Figure 4 shows the imaging 



system configuration of the test environment. 
Although these systems were not included in the 
data gathering, the operators might change the 
DICOM setting to communicate images with the 
experimental MR system as needed. The test was 
conducted on a daily basis between 16 October 1995 
and 16 "January 1996. The acquired data were 
inspected in two ways. First, the completeness of 
each image study hierarchy (i.e. study, series, and 
slice) was verified using the PACS acquisition 
database. The acquisition database contains the 
number of events per study hierarchy which was 
compared to the same information in the scanner 
database. The scanner database information was 
obtained through queries issued by a manufacturers’ 
proprietary software. This information was consid- 
ered to be the actual number of events which should 
be reflected in the PACS acquisition database if no 
images were lost. Second, the acquired images were 
sent to a high resolution (2048x2560) image display 
workstation for inspecting the data integrity of the 
acquired images and text (image header) information 
(see Fig. 4) (9). 

On the console of the GEMS MR system, the 
DICOM C-Store client transfers images using one of 
the following three modes: auto-transfer, auto-queue, 
or manual-transfer. Only one transfer mode can be in 
operation at a time. The auto-transfer mode trans- 
mits a reconstructed MR image slice whenever it is 
made available. The auto-queue mode transfers 
images only when the entire study has been 
completed. The manual-transfer mode allows a user 




Fig. 4. The diagram shows the network connections of the imaging systems and image display workstations in 
our clinical environnment. The systems used in this study involve the PACS acquisition computer, the high 
resolution display workstation and one of the GE MR Signa 5.4 systems. 
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to select and transfer multiple image slices, series, or 
studies. As mentioned earlier, an image series 
covering the body region of interest is clinically 
meaningful. This implies that a complete image series 
must be made available to the clinician as soon as 
possible. For this reason, on the tyIR console the 
auto-transfer mode was set to transfer images to the 
PACS acquisition computer rather than. the auto- 
queue mode. Since the auto-transfer mode transfers 
images whenever they are made available, it is much 
faster than the auto-queue mode. The manual- 
transfer mode was used only when a re-transmission 
was required. 

Statistical results from the preliminary trial were 
compiled using the PACS acquisition database 
tables. Table 1 shows a tabulation of these results. 
There were a total of 475 studies comprising 1886 
series acquired during the preliminary test period. 
The image data and the image header information of 
these studies were intact and could be viewed in the 
high resolution image display workstation. Of the 
475 total studies, 133 (28%) were completely lost and 
29 (6.1%) were partially acquired, meaning one or 
more series or image slices were missing. Upon 
further investigation it was determined that all of the 
completely or partially lost studies, which we refer to 
as the missing studies in this context, were caused by 
an abrupt change in the image transfer mode; while 
images were being transferred to the PACS acquisi- 
tion computer, or sometime just before, the console 
operator changed the transfer mode and/or destina- 
tion computer. We had to inform the console 
operator to manually re-transmit the missing studies 
and resume the auto-transfer mode. We found this 
procedure to be unacceptable in our automated 
DICOM compliant image acquisition design. 

AUTOMATIC IMAGE RECOVERY SCHEME 

Resulting from the preliminary experiment was 
the need to improve the integrity of the acquired 
image data by addressing the problems of the missing 
studies. A method for automatic detection and 
recovery of the missing studies was crucial to the 
success of a DICOM compliant PACS acquisition 
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system. In the following sections, we present how we 
designed an automatic image recovery scheme for 
this purpose. 

Basis for the image recovery scheme 

The automatic image recovery scheme includes 
two tasks: identifying the missing image studies and 
recovering them. These two tasks can be accom- 
plished by using the DICOM Q/R service class in 
conjunction with the PACS acquisition database. 
The operation mechanism of the Q/R service class 
involves three other DICOM services: the Compo- 
site-Find (C-Find), the Composite-Move (C-Move), 
and C-Store. The task of identifying missing studies 
is performed through the C-Find service class by 
matching one of the image grouping hierarchies, such 
as study level or series level, between the PACS 
acquisition database and the scanner database. The 
recovering task follows the missing study detection 
and is carried out by the C-Move and C-Store 
services. In this study, the Q/R operation is 
performed via a server process and a client process 
which run in the MR system and the PACS 
acquisition computer, respectively. 

Design and implementation of the image recovery 
scheme 

Figure 5 shows the flow diagram of the 
automatic image recovery scheme. In the beginning, 
the Q/R client encodes a C-Find query object which 
contains the major information elements such as 
image study level and a zero-length UID value 
(unique identification value) defined by the DICOM 
standard. The zero-length UID prompts the Q/R 
server to return every single UID for the queried 
level. Then, the Q/R server responds with all the 
matching objects in the scanner database according 
to the requested image study level. The content of 
each responded object mainly includes information 
such as a study number and the corresponding study 
UID. The information of each responded object is 
then compared with the records in the PACS 
acquisition database. Those study numbers which 
are in the responded objects but not recorded in the 



Table 1 



Items 


Total 


No. of missing studies due to 
■human intervention errors 


No. of missing 


No. of missing 
studies 


No. of series 
missing images 


Experiments 


no. of studies 
conducted 


Completely 


Partially 


studies manually 
recovered 


automatically 

recovered 


due to grouping 
process 


Preliminary 


475 


133 (28%) 


29 (6.1%) 


162 


0 


0 


Pre-clinical 


259 


49 (19%) 


9 (3.5%) 


0 


5S 


0 





Automated PACS image acquisition • S.L. Lou et al. 



215 



Form C-Find object to 
query according to a 
specified image level (e.g. 
study, series, image) 



Buffer the responded object(s^ 
which include information 
• such as study, series, image 
v number and UID. 



objects 




Perform the imaging 
system database 
lookups and 
Respond matched 
object(s) 



PACS 
acquisition 
database tables 




Sorted 

object(s) 


1 


t 


□ 





Identify the missing ones 
according to the queried 
image level 




Query the other 
image level or 
Wait for 
the next loop 



Fig. 5. A general processing flow diagram of the automatic query/retrieve image recovery scheme. 



PACS acquisition database are considered as missing 
studies. Each of them can be retrieved by issuing a C- 
Move object which is equivalent to an image retrieval 
request. Since the retrieval is study specific, the study 
UID has to be included in the C-Move object 
explicitly. In the scanner, after the C-Move object 
was received by the Q/R server, it relays the retrieval 
request to the C-Store client. As a result, the image 
transfer is actually conducted by the C-Store client 
and server processes. The tasks of identifying and 
recovering missing series are performed with the 
same recovery operation mechanism following the 
study level. The difference between the study level 
and the series level in this recovery scheme is that a 
specific study UID must be encoded in the C-Find 
query object for the series level as opposed to a zero- 
length UID for the study level. 

The records of available images in the scanner 
database and the PACS acquisition database may 
not synchronize. This scenario can happen in the 



current study when its associated series and image 
slices are being generated in the scanner but not yet 
transferred to the PACS acquisition computer. This 
asynchronization can result in incorrect identification 
that the image recovery process mistakes the current 
study as a missing one. A method of avoiding this is 
described as follows: The responded objects from the 
Q/R server are first sorted in chronological order by 
the study creation date and time. If the creation date 
and time of the most recent study is less than a pre- 
defined time-interval compared with the current time, 
it is considered as the study being generated by the 
scanner. Thus, the image recovery process will not 
identify the most recent study as a missing study. 

Concurrent image receiving server 

A very important factor relating to the success of 
the automatic image recovery process is the ability of 
the C-Store server to handle multiple image transfer 
requests concurrently (10). Not only will the C-Store 
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server handle its regular work load of acquiring 
images into the PACS acquisition system, it now 
must be able to handle a C-Store request associated 
with the C-Move service sent by the Q/R server. 
From our experience, with the image recovery 
process running, the probability that the C-Store 
server will receive multiple image transfer requests at 
the same time is high. This indicates that the C-Store 
server running in the PACS acquisition computer 
must consider the design of managing multiple C- 
Store requests in parallel. In order to achieve this 
goal, any factor that may hamper the concurrent 
image transfer must be excluded. For example, the 
locking mechanism of accessing the image transac- 
tion database used in the early version of the CTN C- 
Store server program is not preferred. This is because 
this transaction database can be only accessed by one 
C-Store server process at a time and others may be 
timed-out while waiting. In our implementation, we 
removed the database from the C-Store server 
program because the PACS image acquisition 
database is comprehensive enough to cover the 
image transaction functionality as mentioned in 
section 2.2. 

Pre-clinical experiment 

The integration of the Q/R-based image recovery 
process with the entire acquisition task chain was 
evaluated from 18 March 1996 to 22 April 1996. The 
environment and system set-up were identical to the 
preliminary experiment. In this evaluation period, 
259 studies consisting of 1128 series were acquired 
into the PACS acquisition computer. Among the 
acquired data, there were 49 studies completely lost 
and nine studies partially acquired during the image 
receiving process. They were all identified and 
recovered automatically through the image recovery 
process. Overall, no single series was missed and no 
manual re-transmission was ever required. 

DISCUSSIONS 

Through our research we have found that the 
DICOM standard both helps and hinders the PACS 
acquisition system. On one hand, it significantly 
simplifies the implementation of PACS image acqui- 
sition. On the other hand, due to its open architec- 
tural design, it hinders the integrity of acquired data 
by rendering image studies incomplete. It is our belief 
that this problematic issue can be a major concern 
for DICOM compliant PACS acquisition systems in 
numerous health care institutions around the 
country. We proposed a solution for this problem 
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by integrating the DICOM Q/R service class into the 
PACS image recovery process. 

With the Q/R-based image recovery process fully 
functional and running, the problem of the missing 
studies effectively vanished. During the preliminary 
experiment, when the image recovery process was not 
running, all of the missing studies had to be manually 
re-transmitted, a procedure unacceptable in an 
automated PACS acquisition system. After the 
image recovery process had been implemented, 
every single missing study was automatically detected 
and recovered during the pre-clinical experiment. 
This algorithm guarantees the data integrity of the 
DICOM compliant PACS acquisition system that 
each study will be acquired in its entirety. 

Although the image recovery scheme is effective 
in the PACS acquisition, minimizing the human 
intervention that causes the missing studies is 
important. Two approaches are suggested: The first 
approach is for the manufacturers of the imaging 
systems to incorporate an automatic reset feature. 
When an operator changes the DICOM communica- 
tion setting for an application, the imaging system 
can automatically resume a pre-defined communica- 
tion setting (e.g. PACS transmission) when the 
application is detected inactive. In this way, there 
would be no need for the operator to have to 
manually reset the communication setting. The 
second approach involves an automatic dial-up 
paging scheme similar to the one described in our 
previous work (6). Whenever a missing study is 
detected by the image recovery scheme, a designated 
pager residing on the operator console will be 
activated to remind the operator to reset the PACS 
transmission setting. 

So far, our discussion has focused on solving the 
missing studies caused by human intervention during 
the PACS acquisition. The proposed Q/R-based 
image recovery scheme can also solve the missing 
studies due to hardware and software malfunctions 
during the DICOM image transmission as well. The 
reason is that these missing studies have no 
characteristic difference from those caused by the 
human intervention. Thus, they can be automatically 
recovered provided that the factors causing malfunc- 
tion are removed. 

Although the results from our experiments 
indicate that the NCAITI method in conjunction 
with the PNCS method detects the end of series 
perfectly, the algorithm may miss unusual cases as 
discussed in section 2.3.2. In fact, there is no method 
that can accurately and efficiently determine the end 
of series. However, the indication of the end of series 
may be implemented by the manufacturers of the 
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imaging systems. With such an implementation, not 
only can all the effort spent in detecting the 
completeness of an image series be saved but also 
the overall image acquisition performance will be 
greatly improved. 

One final thought, the proposed Q/R image 
recovery scheme can be used to initiate image 
transfer. In this way, the PACS acquisition computer 
acquires images from the imaging system in a pulling 
fashion. On the other hand, using the DICOM C- 
Store server and client paradigm to perform the 
PACS image acquisition task is considered a pushing 
scheme. For a DICOM compliant PACS image 
acquisition system, the advantage of the pulling 
scheme versus the pushing scheme is that the PACS 
image receiving task will not depend on the DICOM 
communication setting; thus, human intervention 
errors are eliminated. However, the downside of the 
pulling scheme is that the operation workload of the 
imaging system will be increased. 

SUMMARY 

We have integrated an interface mechanism 
based on the DICOM standard which automatically 
acquires images from a GEMS MR scanner into a 
PACS acquisition computer. Since both systems 
follow the DICOM communications protocol, we 
were able to acquire the images quite easily. One 
remaining obstacle is that no end of series can be 
obtained from either the MR scanner or the DICOM 
protocol. Grouping the images into customary sets 
based on their clinical characteristics is an essential 
task of the PACS acquisition process. Two methods 
were introduced for the determination of the end of 
series and their advantages and disadvantages were 
discussed. The design and major functions of the 
PACS image acquisition database were also pre- 
sented. With the DICOM-based PACS acquisition 
setup, we performed a preliminary experiment for a 3 
month period. The results of this experiment indicate 
that over 34% of the studies were not automatically 
acquired into the PACS acquisition computer. This 
result was disappointing and not acceptable. An 
investigation of the missing studies discovered that 
all of them occurred when the transfer mode and/or 
the transfer destination were changed and not reset 
again. A DICOM Q/R service class was implemented 
with the PACS acquisition database to detect the 
missing studies and automatically recover them. The 
PACS image acquisition system with the image 
recovery scheme was evaluated through a two- 
month pre-clinical experiment. The statistical results 
of this experiment show that 259 studies were 



automatically acquired and not a single one was 
missing. This indicates that the Q/R-based image 
recovery process effectively eliminated the missing 
studies from the PACS image receiving process. 
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T wo major events have transformed 
the high technology and health care 
industries during the past several 
years: the rapid development of 
multimedia technology and health care reform. 
Multimedia software and hardware developers 
have largely targeted the consumer market, shap- 
ing the development of computing, communica- 
tion, and visualization technologies to emphasize 
entertainment and education. The health care 
industry, on the other hand, is going through 
structural, institution-level changes that will alter 
how health care is delivered in the coming years. 
Such reform highlights the need for high tech- 
nology like multimedia to facilitate the shift to 
such cost-effective methods as wide-area net- 
works, databases, and telemedicine. 

Health care is a triad emphasizing patient care, 
education, and research. Patient care focuses on 
quality care with minimum cost; education focus- 
es on training future health care professionals and 
creating patient and community health awareness; 
research focuses on new discoveries in diagnostic 
and therapeutic methods and procedures. 
Multimedia technology can facilitate a synergism 
among these components. In the health care 
industry, major advances in the past ten years 
include the development of hospital information 
systems (HIS) and picture archiving and commu- 
nication systems (PACS). HIS manages patient data 
flow including demographics, billing, scheduling, 
reports, and statistics. PACS, on the other hand, 
was developed- to help clinicians and researchers 
manage medical images. The integration of HIS 
and PACS opens the door for the health care 
industry to use multimedia technology widely. 



This special issue singles out developments and 
applications of multimedia technology for the 
health care industry. The article "Networked 
Multimedia for Medical Imaging" by Wong and 
Huang describes the state of the art of HIS and 
PACS and their use for patient care. It also intro- 
duces a novel methodology that integrates HIS 
and PACS in the networked multimedia environ- 
ment for better health care delivery, education, 
and research. 

Thoma and Long's article, "Visible Human 
Images," describes a type of medical image devel- 
oped by the National Library of Medicine, a US 
government agency. The images are now ready for 
dissemination to the public for education and 
research purposes. The authors describe the tech- 
nology behind the project and note that speed 
and ease of access and low cost are the keys to 
public awareness and acceptance. 

The article "Opportunities for Visual 
Computing and Communications in Healthcare" 
by Kitson, Malzbender, and Bhaskaran discusses 
technology industry efforts to develop multi- 
media components for health care applications. 
Interest in bringing multimedia applications to 
health care will continue to grow rapidly and will 
surely benefit patient care, reduce costs, and 
advance research and education. These three arti- 
cles unveil general multimedia requirements in 
health care and forecast trends in multimedia 
technology development. Read on, and explore 
the future of multimedia in medicine. MM 



H.K. Huang is a professor and vice 
chairman of the Department of 
Radiology and director of the 
Radiological Informatics Laboratory 
at the University of California, San 
Francisco; he also holds a joint 
appointment in the UCSF/UC-Berkeley Bioengineering 
Graduate Program. Huang implemented a PACS at 
UCLA in 1991 and a second-generation PACS at UCSF 
in 1994. His current research interests include medical 
multimedia, high-speed medical image transmission, 
and image content indexing and retrieval. 




Contact Huang at the Department of Radiology, 
University of California, San Francisco, 505 Parnassus 
Ave., San Francisco, CA 94143-0628, e-mail 
bemie_huang@radmacl .ucsf.edu. 



1 070-986X/97/S1 0.00 C 1 997 IEEE 



138 



23 








Networked 
Multimedia for 
Medical Imaging 

Stephen T.C. Wong and H.K. Huang 
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The development of 
biomedical imaging 
has revolutionized 
medical practice and 
health care delivery. 
The next wave of 
change, however, 
will come from 
merging medical 
imaging with 
multimedia 
technology. This 
article discusses the 
necessary 
information 
infrastructure and 
supporting 
technology for this 
evolving field. We 
describe four types of 
clinical applications 
to illustrate the use 
and versatility of 
networked 
multimedia systems 
for image-assisted 
medical care. 



T oday's medical images come from a 
broad spectrum of imaging sources 
and contribute important information 
in a wide range of medical contexts. In 
the United States, about 30 percent of diagnostic 
imaging examinations are taken directly in digital 
media. These include X-ray computed tomogra- 
phy (CT), magnetic resonance imaging (MRI), 
ultrasound (US), positron emission tomography 
(PET), single photon emission computed tomog- 
raphy (SPECT), digital subtraction angiography 
(DSA), and computed radiography (CR). The other 
70 percent of examinations — skull, chest, breast, 
abdomen, and bone — employ conventional X-rays 
and microscopy. Digitizers such as laser scanners, 
solid-state cameras, drum scanners, and video 
cameras can be used to convert X-ray films and 
microscopic image pictures into digital format for 
processing. 1 The amount of digital medical image 
data captured per year in the United States alone 
is estimated on the order of petabytes— 10 IS 
bytes — and increases rapidly every year. 

These modalities have revolutionized how we 
acquire patient images and even permit evalua- 
tion of various biologic processes and events as 
they occur in vivo. For example, the emergence of 
MRI methods and improvements in CT acquisi- 
tion techniques have led to safe and convenient 
ways to visualize fine anatomical structures such 
as blood vessels. Nuclear medicine scans such as 
PET and SPECT generate functional images of 
blood flow and metabolism essential to diagnosis 
and to research on various organs such as the 
brain, heart, liver, bones, and kidneys. 

Medical images thus form the cornerstone of 
the patient record and lie at the heart of diagno- 



sis, therapy, and follow up. While physics and 
mathematical research have contributed greatly 
to medical imaging and will continue to do so, 
new development efforts are shifting from gener- 
ation and acquisition to postprocessing and man- 
agement of medical images and associated data. 
Medical imaging information types include 

I still images, whether 2D such as mammograms 
or 3D such as CT scans; 

I pictures, such as image slices from a biopsy; 

I moving images, whether motion video such as 
ultrasound or cine images such as cardiac MRls; 

■ structured text such as image headers; 

I plain text (unformatted) such as diagnostic 
reports; 

I sound such as a physician's dictation; and 

I graphics such as a geometrical model of the 
brain or annotation overlaid on an image. 

Several factors drive this shift toward multi- 
media applications in medical imaging: 

I increasing market demands for cost-effective 
health care delivery, 

I high capital investment in imaging device 
development, 

I the need to organize and store large amounts 
of multimodal image data and complement 
them with relevant clinical data, 

■ improved performance and falling costs of 
computers and communication systems, and 

I the ability to extract rich information embed- 
ded in images. 

This last factor exceeds the sum of individual 
imaging technologies and generates computer- 
based multimedia applications for a wide range of 
diagnosis and therapeutic procedures. Such appli- 
cations use the computer to integrate, control, 
and correlate several types of clinical information 
that can be represented, stored, transmitted, 
processed, and manipulated digitally. 

Fluckiger divided computer-based multimedia 
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applications into two categories: local and net- 
worked multimedia. 2 This classification also 
applies to medical imaging. 

Local multimedia applications run on stand- 
alone systems and use no remote resources. An 
example is a series of self-guided teaching files dis- 
tributed by the American College of Radiology 
(ACR). Such computer-based education programs 
ran on CD-ROM-equipped PCs and interactively 
take residents and practitioners through the steps 
of diagnosing selected imaging cases. 

Networked multimedia applications provide 
services via remote systems communicating across 
networks. Picture archiving and communication 
systems (PACS) used in many digital radiology 
departments acquire images directly from hospi- 
tal scanners, consolidate image files in a central 
on-line archive, and transmit the files to remote 
high-resolution display stations for review. 3 

We focus here on networked digital multi- 
media in medical imaging. Networked multi- 
media can support genuinely networked 
applications or a client-server set-up. Genuinely 
networked applications such as telemedicine and 
teleradiology offer healthcare services across dis- 
tances. Even when supporting applications for 
which stand-alone systems might suffice, a client- 
server model provides a cost-effective means to 
pool resources such as large-scale multimedia stor- 
age capacity into servers remotely accessed by 
client medical workstations. 




Figure 1. The networked 
multimedia medical 
imaging framework and 
its logical components. 
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A networked multimedia framework 

To develop new networked multimedia appli- 
cations, we first need to build a new infrastructure 
that can support them. Figure 1 shows a logical 
framework to support multimedia medical imag- 
ing. Its components include data sources (PACS 
archives and other image data stores), postpro- 
cessing, communication, visualization, image 
database management, and client application 
workstations. 

Data sources 

Patient data come from PACS and associated 
medical information systems. Data types include 
patient demographics, medical history, still and 
moving images, diagnostic reports, voice dicta- 
tion, video, and other clinical data. Most of these 
are archived as flat files (such as PACS images) or 
in nonrelational textual databases (such as the 
hierarchical data model used in most hospital 
information systems). These can be archived and 
shared using recognized data exchange standards 



and protocols such as Dicom (Digital Imaging and 
Communication in Medicine) for medical images 4 
and HL7 (Health Level 7) for clinical text. 5 With 
raw images and other medical information in dig- 
ital media, we can postprocess, create, and present 
image contents for new multimedia medical imag- 
ing applications. 

Table 1 lists the average amount of data per 
examination generated by a spectrum of medical 
imaging technologies. The image size and num- 
ber of images taken in one patient examination 
vary by modality. Except for digitized electronic 
microscopy and digitized color microscopy, which 
are pathologic and histologic images of micro- 
scopic tissues, we classify the modalities as radio- 
logic images and use them for diagnosis and 
treatment planning. In contrast with most other 
true-color biomedical image types (such as DCM), 
a radiologic image is monochrome, and each radi- 
ologic examination follows a well-defined proce- 
dure. For example, one X-ray CT examination 
(about 40 image slices) with each image slice size 
of 512 x 512 x 12 bits generates 20 to 40 Mbytes, 



140 



25 



April-)une1997 






IEEE MultiMedia 



whereas one digital mammography image may 
generate 32 Mbytes of data. (X-ray films requiring 
higher resolution can be digitized using 4,000 x 
4,000 x 12-bit digitizers.) Note that dynamic med- 
ical images such as digital subtraction angiogra- 
phy for cardiac catheterization generate multiple 
runs of video data per study. Each run lasts a few 
seconds to tens of seconds and can generate half 
a gigabyte of digital data. 

Image postprocessing 

Images ignore semantic contents and exist as 
bitmaps consisting of individual pixels, each 
coded with a fixed number of bits. Imaging spe- 
cialists must interpret medical images qualitative- 
ly. For the past three decades, the medical image 
processing community has actively pursued effi- 
cient algorithms that can extract and quantify 
semantic objects from plain image files. 6 We can 
readily integrate established algorithms into the 
networked multimedia framework to process 
existing digital images for new applications. 
Typical tasks include image segmentation, texture 
analysis, content extraction, and image registra- 
tion. These can be performed automatically or 
interactively depending on the difficulty of seg- 
menting and extracting semantic structures. 

Communication 

Enabling technologies with high-resolution 
images include networking, data security, image 
compression, and groupware. Networking 
employs asynchronous and synchronous com- 
munication modes. The former includes image 
acquisition and exchange, while the latter refers 
to group teleconferencing and interpersonal com- 
munication. Most hospital image networks use 
Ethernet, Fiber Distributed Data Interface (FDDI), 
or Asynchronous Transfer Mode (ATM). Medical 
image communication occurs almost universally 
over transmission-control protocol/Internet pro- 
tocol (TCP/IP), but real-time protocols for ATM- 
based multimedia networks remain an unresolved 
issue. 

We must consider security when implement- 
ing both data integrity and access protocols. 
Wong recently discussed medical image security. 7 
After postprocessing, mechanisms such as key- 
based encryption and digital time stamping will 
ensure the completeness and authenticity of med- 
ical images. Data access systems determine who 
can access what type of data and when. 

Image compression reduces digital medical 
image data volume and achieves a low bit rate 



without perceived loss of image quality. Demands 
for transmission bandwidth and storage space, 
however, continue to outstrip existing techno- 
logical capabilities. Furthermore, lossy coding 
techniques for clinical use raise many complex 
legal and regulatory issues. Wong focused on 
progress in lossless and lossy medical image com- 
pression and the legal challenges of using lossy 
compression for medical records. 8 

Groupware designates the computer-based 
software and systems that support collaborative 
work among remote sites. 9 Multimedia medical 
imaging requires shared computer workspace that 
can accommodate 3D images of 10 to 12 bits per 
pixel, videoconferencing that can display high- 
resolution medical images (see Table 1), and the 
ability to access remote medical databases. 

Data visualization 

The need to improve tools for volume visual- 
ization of medical images has become a bottle- 
neck in multimedia medical applications. This 
affects the combined interpretation or correlation 
of 3D anatomic and physiologic or metabolic 
data. The challenge lies not only in handling large 
medical images but also in representing complex 
relationships among the data. Current visualiza- 
tion techniques for biomedical data analysis 
include surface-rendered anatomical displays with 
rotation and shading, volume-rendered cutouts 
with emphasis on particular objects, transparent 
surfaces within surfaces with color shading and 
rotation, superimposition of multiple image data 
sets of the same organ, and reprojection tech- 
niques such as maximum-intensity projection 
(MIP) and weighted-integrated projection, which 
use various weightings of pixels of interest. 

An example already in routine use is the cine 
CT angiogram, visualized in 3D by projecting at 
multiple different angles to form a sequence of 
images that can be played back in "cine" mode to 
simulate the rotation of blood vessels. Recent 
advances in 3D graphics accelerators also let us 
volume-render tens of megabytes of image data on 
the fly; this is critical for real-time image-guided 
therapy and surgery procedures. 

Medical image database management 

This middleware component integrates and 
organizes images and other patient data, selected 
image features, and clinical decision rules into a 
database management system (DBMS). It supports 
multimedia database management, content-based 
indexing, formatting and distribution for visual- 
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ization, and image processing. A medical image 
database server thus pools hospital network 
resources in a uniform and easily accessible man- 
ner for client application development. Because 
no existing DBMS can manage medical images 
and their contents, we must develop such a server 
using customized object software or some mid- 
dleware specifications such as the common object 
resource broker architecture (Corba) that will 
interact with the underlying databases. 

Application client workstations 

Once the infrastructural components are in 
place, we can design client workstations that inte- 
grate imaging applications for clinical services, 
research, and education. The application software 
also optimizes workstation design for information 
retrieval, presentation, and visualization with 
minimal user effort. User acceptance depends on a 
well-designed graphical user interface (GUI). GUIs 
for today's medical imaging workstations are 
mostly Unix-based. Soon, however, the World 
Wide Web and associated programming languages 
such as Java and HTML combined with greater PC 
processing power will let us deploy Web browsers 
and PCs even for demanding medical imaging 
operations. 10 Because most of its components are 
sharable and reusable, the networked multimedia 
framework enables rapid prototyping and reduces 
development costs. 

Multimedia medical imaging applications 

For the past three years, we have been devel- 
oping a networked multimedia system testbed. 
We started with the existing storage and net- 
working infrastructure of the UCSF hospital inte- 
grated picture archiving and communication 
system (HI-PACS). We then implemented a mid- 
dleware image database server (IDBS) to manage 
multiple medical image databases and interact 
with data visualization, postprocessing, and other 
communication utilities. 11 ' 12 HI-PACS connects 
imaging scanners of the UCSF Medical Center and 
affiliated hospitals in the San Francisco Bay Area 
using 155-Mbps ATM wide and local area links. 
The system stores automatically generated images 
in its 1.3-terabyte optical jukebox together with 
relevant patient data from radiology and other 
hospital information systems. 

This article focuses on some of our ongoing 
applications to demonstrate the uses and benefits 
of such a multimedia information infrastructure. 
We discuss four medical imaging applications built 
on the networked multimedia information system 




actioaangj>gra|D^ ; . 



t ., 

% -MRI ^-magnetic- resonance jrnaging^-pj.t^- 
MRS 1 magnetic resonance spectroscopy ' ; 

L MSI -^magnetic source Imaging - 

- PET '■' positron emission tomography ’ 
SPECT single photon emission computed 
'.. y tomography . 

US ultrasound 



testbed. For each application we present the under- 
lying hypothesis, conventional methods and their 
shortcomings, the new approach using networked 
multimedia techniques, and specific components 
required. 

Bone age assessment 

Physicians frequently perform bone age assess- 
ment in pediatric patients to evaluate the normal- 
ity of their skeletal growth. 

Hypothesis. Bone age can be assessed by auto- 
matically comparing a left hand and wrist digital X- 
ray image with a comprehensive hand digital atlas. 

Conventional approach. Physicians perform 
radiological examination of the child's left wrist 
because it offers simplicity, minimal radiation 
exposure, and multiple ossification centers to 
evaluate maturity. This provides important infor- 
mation for the diagnosis and management of 
endocrine disorders and diagnostic evaluation of 
metabolic and growth abnormalities, acceleration 
or decrease of a variety of syndromes, malforma- 
tions, and bone dysplasias. Doctors also actively 
consult this record in planning an orthopedic pro- 
cedure because childhood is the best time to cor- 
rect scoliosis or limb length abnormalities. 

Conventional bone age assessment compares a 
hand radiograph with a reference set of hand X- 
ray film data developed in the 1930s and pub- 
lished in the Greulich and Pyle atlas. 13 This 
method, although used for more than thirty years, 
is inaccurate and not fully applicable to today's 
children because of its limited sample size and the 
data set's regional nature. 
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Figure 2. Query user 
interface developed for 
bone age assessment 
application. We have 
changed all names for 
confidentiality. The 
image on the right is the 
hand digital radiograph 
to be assessed. 



Networked multimedia approach. We need 
to establish a digital atlas with a large standard set 
of normal hand and wrist radiographs and associ- 
ated measurements. This will form the basis for 
computer-assisted bone age assessment. The tasks 
include 

1. developing a new large reference set of clini- 
cally normal hand images that will serve as a 
digital hand atlas; 



2. providing computer-assisted assessment of 
skeletal development by extracting relevant 
features from a patient image and comparing 
these with corresponding values extracted 
from the digital atlas, thus creating an updated 
set of reference images; 

3. creating a database system that organizes atlas 
images, extracted features, patient age and sex, 
and clinical findings; and 



4. allowing remote access to the bone image 
database from physician workstations. 

The multimedia components include (see 
Figure 1): 

Data source: Use computed radiography (CR) 
images and related data from the PACS image 
file archive to generate the atlas. 

Postprocessing: Develop edge enhancement 
and morphological filtering software to extract 




parameters from CR images for bone age assess- 
ment. 14 The features of interest include distal, 
middle, and proximal phalange lengths, epi- 
physeal and metaphyseal diameters, and other 
carpal and metacarpal bone geometry. 

I Database management: Implement a relation- 
al database to organize and store the digital 
atlas and associated data. The IDBS manages 
this bone age database. 12 

I Communication: Use existing PACS networks 
to connect the IDBS, bone age database, PACS 
archive server, and client workstations. 

I Application software: Design a client worksta- 
tion that lets users query the bone age database 
and evaluate the computerized bone age assess- 
ment. Query can be by image content (hand 
bone age or epiphyseal and metaphyseal diam- 
eter ratio), patient attributes (name, age, and 
examination date), or a combination of image 
and text features. 

Figure 2 illustrates the implemented GUI. The 
sliders in the bottom left "Query by Image 
Attributes" window specify the range of image 
attributes for data retrieval. The IDBS returns a list 
of five patients (upper left window) and thumb- 
nail pictures satisfying the combined image and 
patient attribute constraints. The user can then 
click on any thumbnail image to retrieve and ana- 
lyze the original hand radiograph. 

Temporal lung nodule analysis 

Quantifying nodule features, especially size 
and temporal progression, permits clinicians to 
assess the effectiveness of therapeutic measures or 
determine the characteristics of particular lesions 
in the lung. For instance, the volume-doubling 
time interval of a solitary nodule indicates 
whether the nodule is benign or malignant. Also, 
determining whether a small nodule located deep 
in a lung is benign involves detecting whether its 
volume doubles within a short (less than 30 days) 
or long (more than 490 days) period. For metasta- 
sis, nodule volume variations reflect drug treat- 
ment efficacy. An effective drug reduces or halts 
cancer cell division; the volume of the malignant 
nodules thus should not increase during the treat- 
ment period. 

Hypothesis. Physicians can track and analyze 
the progression of lung nodules over time with 
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